Arrogance of Joseph Smith
History of the Church 6:408-409; “Come on! ye prosecutors! ye false swearers! All hell, boil over! Ye burning mountains, roll down your lava! for I will come out on the top at last. I have more to boast of than ever any man had. I am the only man that has ever been able to keep a whole church together since the days of Adam. A large majority of the whole have stood by me. Neither Paul, John, Peter, nor Jesus ever did it. I boast that no man ever did such a work as I. The followers of Jesus ran away from Him; but the Latter-day Saints never ran away from me yet.”
Deut. 34:10; “And there arose not a prophet since in Israel like unto Moses, whom the LORD knew face to face.”
Deut. 28:66; “And thy life shall hang in doubt before thee; and thou shalt fear day and night, and shalt have none assurance of thy life.”
At the time of Smith’s boastful speech people had indeed left the fold. As a matter of fact this was an address towards the dissenters of the Church given on Sunday, May 26, 1844; just two short months later Smith would be killed.
In 1838 Smith and company had ex-communicated about 50 people for various reasons. A large group of people were ex-communicated that spring for accusing Smith of polygamy amongst other nefarious deeds. In that group the entire Whitmer family was thrown out of the Church and most of those were the original witnesses to the Book of Mormon.
http://lifeafterministry.com/2011/10/16/book-of-mormon-witnesses
But Joseph Smith never claimed that no one had left the church, or were ex-communicated. He said “A large majority of the whole [had] stood by” him, and he was completely accurate.
50 people out of several thousand is a very small number.