Which ebook format do books from the iBooks Store use

You have to use iTunes.

Open iTunes, click on your device and then choose the Books tab. There you can select which books you want to sync to your iOS device.

It appears to sync your pages placement too.

In short, the iBooks application and its behavior were kind of a big surprise to everybody. Before I opened it, Mavericks seemed great.

The iBooks application seems positioned to deal only with purchased (DRM) books. This includes your ePubs and PDFs. It wasn't announced, but it seems to be the case. Also the iBooks application on the desktop isn't intended to be the means to sync to your iDevices. When you sync your device, the iTunes "Books" selections will come from the new iBooks application.

All of your not-purchaed ePubs and PDFs that you used to store and sync in iTunes, well now you have to use something else, like Calibre. Whether or not this continues to be the case is unknown. It's possible that the new iBooks app is an attempt to distance Apple from non purchased books.

I spent a day putting all my non-purchased books into Calibre and it works fine. It's not seamless with the syncing, especially if you used to do it over WiFi. For the moment that capability is not available. Calibre has multiple methods for copying to devices, and you will definitely need to do a little digging in the documentation.

So for the moment, you have two separate methods of syncing books: iBooks and iTunes for purchased, and Calibre (or whatever) for non-purchased.

When you start iBooks for the first time, your iTunes books will be transferred to a folder buried out-of-sight. Most of the ePub metadata will be in the ePub file, but not so for PDFs. If you were lucky you copied your stuff out of iTunes before using iBooks. I wasn't so lucky.

BTW, any PDFs that are music booklets (or in my case, sheet music) can stay with the album in iTunes. However they are media type "Music".