What does Peter mean by saying that judgment starts in the house of God?
Judgment of All
The Bible says that all of us will be judged by God but the critical difference in this judgment is whether God’s brought a person to repentance and faith or if they’ve rejected Jesus Christ; either way, both the saved and the lost are judged, however, the person who has trusted in Christ have had their sins atoned for in Jesus Christ and have no further worry about God’s judgment after death or at Christ’s appearance (Heb 9:27). But for the person who refused to acknowledge Jesus Christ as Lord and Savior, they too will be judged, but they will have to pay for their own sins, and of course that payment is unending (Rev 21:8). That’s exactly why it’s so crucial to remind them of Jesus’ words: “Whoever believes in the Son has eternal life; whoever does not obey the Son shall not see life, but the wrath of God remains on him” (John 3:36), therefore, we are to “Repay no one evil for evil, but give thought to do what is honorable in the sight of all” (Rom 12:17, and why should we not repay evil for evil? Because we are told, “never avenge yourselves, but leave it to the wrath of God, for it is written, “Vengeance is mine, I will repay, says the Lord” (Rom 12:19). It is God Who judges all, but judgment begins in the house of God, or with the church. It’s not for us to judge those outside of the four walls of the church but only those inside (Matt 18:15-20). This is why we must tell them of the coming wrath of God and that they must repent and believe, just as Jesus required (Mark 1:14-15), or they will perish (Luke 13:3-5). Then, we must leave it up to the Holy Spirit to take that message to their heart because God is the true evangelist (John 1:12-13, 6:44).
The House of God
When we read about judgment starting with the house of God, we understand that this is the universal church, which is the church that Jesus purchased with His own blood. He is the Head of the church, and He expects the church to discipline its members when it is necessary, and the pastor is not above discipline, or in some cases, being fired. When a pastor begins to drift away from the Christo-centric, Bible-centered preaching and teaching, he has lost his source of power (Rom 1:16; 1st Cor 1:18), and the church as a whole must address this or it ceases to have Jesus’ blessing. It could be what Jesus was talking about in the Book of Revelation, where their lamp stand or candlestick is taken away (Rev 2:1, 5), or worse yet, have Jesus say to them, “I know your works: you are neither cold nor hot. Would that you were either cold or hot! So, because you are lukewarm, and neither hot nor cold, I will spit you out of my mouth” (Rev 3:15-16). In other words, lukewarm churches make Jesus’ sick to His stomach, figuratively speaking of course, so every one of us is accountable to God and will have to stand before Him to explain our dereliction of duty and negligence of church discipline, absence of the church ordinances (e.g., Communion), and good, expository, or verse by verse Bible preaching and teaching. If you lose any one of these links, the whole chain breaks down. Jesus was the first to teach about church discipline (Matt 18:15-20), but the Apostle Paul also believed in it, since allowing even a little leavening in the church, can infect the whole lump (the local church). What do we do with the unrepentant sinner in the church? Paul says, “you are to deliver this man to Satan for the destruction of the flesh, so that his spirit may be saved in the day of the Lord” (1st Cor 5:5). The so-called “hedge of protection” is removed, and Satan or his demons are free to buffet them, but the idea is not to destroy them but to bring them to repentance and return them to the fold. Late in Paul’s ministry, it was “Hymenaeus and Alexander, whom I have handed over to Satan that they may learn not to blaspheme” (1st Tim 1:20), so it’s for the good of the church. Besides, God disciplines us…and it is purely motivated out of love (Prov 3:12; Heb 12:6).
Those Who Don’t Obey
I believe that it takes the Word of God from a person of God with the Spirit of God to make the children of God, and all for the glory of God. It’s all about God, but God does use us as a means to achieve His ends. We cannot judge others outside of the church for several reasons. For one thing, they may be called by God at a later time, or they may be one who God is not calling, so we don’t know who is or who is necessarily saved. All we can do is unleash the power of the gospel (Rom 1:16; 1st Cor 4:18) and trust God with the results. The seed takes time, more time with some than others, but it is God Who gives the increase (Acts 2:47), so just be faithful witnesses, and leave the results up to God. What did God’s Word do to us? In God’s infinite mercy, “when we are judged by the Lord, we are disciplined so that we may not be condemned along with the world” (1st Cor 11:32), but think of the lost who have not yet trusted in Christ. You can almost hear the passion in the Apostle Peter’s voice where he writes, “For the time is come that judgment must begin at the house of God: and if it first begin at us, what shall the end be of them that obey not the gospel of God?” (1st Pet 4:17 KJV)? If Jesus had to endure excruciating agony on our behalf, then what will come of those who reject their only hope, or “If the righteous is scarcely saved, what will become of the ungodly and the sinner” (1st Pet 4:18)? That is a very sobering question, and it should make us bolder witnesses for Christ  because we know what awaits the lost.
I do not think its right to go up to people who are living in sin, or are clearly sinners, and tell them that they’re condemned to hell. Instead of being in their face or thumping them over the head with the Bible (with Bible verses), we should just be “prepared to make a defense to anyone who asks you for a reason for the hope that is in you; yet do it with gentleness and respect” (1st Pet 3:15). Gentleness and respect is the way we would wish to be treated, and so we should give honor to others to whom honor is due. Let us all obey the new commandment Jesus gave, and that is, “that you love one another: just as I have loved you, you also are to love one another. By this all people will know that you are my disciples, if you have love for one another” (John 13:34-35). It’s not our convincing arguments, or our weighty apologetics arguments that makes the difference. It is the Word of God and the Spirit of God, but the love of God certainly helps. Jesus said that’s how they’ll know who are and who aren’t His disciples. It’s not by condemning them and judging them, even before they’ve been exposed to the gospel. Jesus said, “God did not send his Son into the world to condemn the world, but in order that the world might be saved through him” (John 3:17), so neither should we condemn or judge the world. God’s throne is a one-seater, and I’m certain He doesn’t need our help.
Take a look at this related article: Should Christians Judge Outside of the Church? 
Resource – Scripture quotations are from The Holy Bible, English Standard Version® (ESV®), Crossway Bibles. (2007). ESV: Study Bible : English standard version. Wheaton, Ill: Crossway Bibles. Used by permission. All rights reserved.