Hi all, I’m looking at moving over to qdb cloud. Initially, I’m thinking of two cloud instances: one being a production server with a heavily guarded ingestion process. The other being more open to R&D….is there a way to query one and insert on the other ?

hey Orrin, you will need to double-publish to two instances for now. With ver 7 we just released you can use WAL tables. These tables will produce least to no pressure on your publishing infrastructure. So if you R&D box is busy querying, publishing to that box will lag a bit owing to WAL being on the box but not yet visible to reads.

Thanks vlad. What you say makes sense although may not solve the central scenario whereby we would tend to be pushing data to prod in daily snapshots and then ad-hoc need to have that data on the R&D box (maybe 10 times in a day if we keep destroying it, but maybe not for a month…depends on research task) It’s not a big deal to pull onto a local machine and push back though to other cloud instance

we have similar use case btw ... I was thinking that something like periodic snapshot of prod db and from that snapshot creates / replaces automatically the secondary (R&D) instance would solve this ... we do this ourselves with open source version but if we move into cloud at some point this would be something we would miss as well

Cloud allows for automatic incremental snapshots. Snapshotting takes a few minutes (depending on workload), but has no effect on your instance. Restoring a snapshot into a new instance takes ~2 minutes (the standard time it takes AWS to create a new instance). This could work for you, with the caveat that restoring is, as of today, a manual task

agree that this would be fine but we would need it in automated way if we are switching to cloud at some point (will depend on how cold storage works etc 🙂 ... so just mentioning that some automation around this has use cases that probably other people would use as well

Another option that might work for you is just running a cronjob doing an API /exp in the prod instance and an /imp into the other one.

understand

we can automate this, but perhaps in the next version or versions (really near future). WAL tables function such that we create WAL files as an intermediate step. After they are applied they are deleted to save space. Rather than deleting we will zip them and upload to S3. When you want an R&D db to catchup files are downloaded and applied. Replication will have that process triggered on each commit or thereabouts, fully automatic. But an easier to implement solution would be periodic pull from S3.