File size does matter to a certain extent in how your book will perform in the reader.
Pages with lots of assets that have large file sizes may tend to slow down in the reader or take a little while to load.
The best rule of thumb is to try to keep the file sizes of your assets as low as possible without affecting the quality of the asset.
For images, use a .png for any image that contains transparent pixels (such as a character cut out of it's environment).
Use a .jpg for any image that does not contain any transparent pixels (such as a background image). Where a .png may be 2 or 3 megabytes in size, that same image will be about 500k as a .jpg and that savings will help your page perform better without reducing the quality.
For sound, try to loop sound where you can and fade out sound where you can. Instead of just playing a 2 minute long piece of music, try to make the melody loop, then move to the next looped melody at a tap point. This not only reduces file size dramatically (a 2 minute piece of music can be 3 meg or more in size depending on the file), but it also makes the reading experience more interesting as the music and sounds build through user interaction with the tap points.
Overall, try to keep any one asset to below 1 meg if possible, and in the low 100k's if possible.
This will make your book load faster and perform better in the reader.