Maximum size WARC files for ReplayWeb viewer in Starter

  • 22 May 2024
  • 2 replies

Badge +1

Hello, we are currently researching using Starter for the archving of a not for profit community website.  When we produce, with a crawler, a test crawl of the site producing a 1GB file, the WARC file is shown correctly in the viewer.

When we do a full crawl , just over 2GB , it doesn’t render.

Is there an upper limit to the allowable size of WARC files that you know of?

thank you

2 replies

Userlevel 4

It’s not so much that there is a fixed maximum size, as that expects a WACZ so it has to be transcoded first, and if that takes too long the render request will get timed out.

If you can get your crawler to generate a WACZ and ingest that into Preservica instead you might get a better outcome.

Badge +1

That is very interesting, thank you Richard.  We will try again with a WACZ: we had no luck with that before, but it’s worth trying again.