Archive service integration

submitted by [deleted] from (self.IdeasForSaidIt)

So a link's content can never be lost. Discussion prompting the idea.

Approaches:

  1. archive.org + a saidit bot: the reddit approach, a user must click to archive
  2. an archive service run by saidit
  3. other
  4. SaidIt decentralized archival backbone

Research tasks:

  1. Does archive.org show the ip of the archiver?
  2. Are there alternatives to archive.org?
  3. Are there open source archive.org equivalents?
all 11 comments
sorted by:
top
newinsightfulfun

[–]Mnemonic 3 insightful - 1 fun3 insightful - 0 fun4 insightful - 1 fun -  (8 children)

1 or 3

2 is too much work regarding law and copyright and whatnot, especially with the EU cracking down. I think is 'the law' is ever going to go after saidit an archive service alone is enough.

Let alone the case of someone feeding it 'illegal content' (like the stuff noody wants to archive).

In the case of 1 I would maybe think of a moderator having the option to archive it? Or some other restriction upon it, because even if it's a 'click to archive' option there are always people that go 'everything I post is worth the archive'.

[–][deleted] 2 insightful - 1 fun2 insightful - 0 fun3 insightful - 1 fun -  (1 child)

Thanks for the feedback. Yeah 2 is probably puts SaidIt is a compromising position.

a moderator having the option to archive it?

That gets harder to implement but yeah we could probably do it that way. Is it worth us worrying about overwhelming archive.org? Any activity from SaidIt is bound to be minuscule compared to how much they are archiving daily.

[–]JasonCarswell 2 insightful - 1 fun2 insightful - 0 fun3 insightful - 1 fun -  (0 children)

I wouldn't worry about Archive.org. It's in the Presidio in SF, a former military base. Nice folks work there but I never met the management. They've got enough angel funds to last forever.

[–]JasonCarswell 2 insightful - 1 fun2 insightful - 0 fun3 insightful - 1 fun -  (4 children)

IMHO Your take on it has a faulty foundation, in part due to the way it's phrasing was proposed. At least, according to the linked discussion we were having.

Approach #2 should not say "an archive service run by SaidIt".

Rather it should say "a decentralized archive service discussed in a SaidIt subsaidit".

It may be called the "DIY_WebScrape_Archive" subsaidit. Within it everyone may discuss ideas and solutions, tips and tricks, recommend gear, share code, etc. SaidIt won't be running it, just hosting a discussion. The people will run it. There is no leader. I am Spartacus. Starting with "their" archive will be good but collectively we will figure out how to build our own decentralized archives with various protocols and various levels of commitment (ie. RSS or text-only archives are smaller than with-images are smaller than with-media archives).

If "we" run it then the only thing we have to fear is if "they" decide that we can't discuss it here. That's when the freeshpeaches go rotten. Before that happens we may be able to relocate to other safe havens like QxR where they discuss x265 torrents. But I suspect our regular speech is more of a problem than decentralized archival copyright issues, especially when "they" are doing it both secretly and at Archive.org without our permission - unless some dangerously damning stuff gets backed up and they feel compelled to go nuclear.

Separate from SaidIt this federation of archivists of individuals would be responsible for their own archives. They would have their list of trusted sources, including but not exclusively SaidIt.

I don't know exactly how this works. Obviously if I post "illegal content" on SaidIt I can be banned. That's not on SaidIt. Unless SaidIt does nothing about it. I don't know if there's a reasonable time limit or what the exact circumstances are, but I'd assume the same might be applicable to all members of a federation. If Bob is away on vacation or is too busy to check his email notifications that someone's been putting "illegal content" on his server, not only are the other members going to discuss this but they're also going to block Bob until he get's it together. If Bob is a weak link too often he'll be on special watch or ejected or brought down or become a crime boss in the underworld.

Also there may be ways to encrypt all the data others archive on your server like the old dark net FreeNet or I2P where you don't know what you have and you can't ever see it, while your backups on other servers can't be seen by anyone but you. Further still, you may only get incomplete parts of files, fragmented across many.

Now this is a hypothetical idea: There may be a way to authenticate some files. If several people agree the hash and the file are legit then it may not need encryption and it can be openly shared as "safe". Of course, with the equivalent of a 51% hack you may be compromised but at that point you're dealing with bigger organized fish that have somehow infiltrated your circles of trust.

This decentralized archival thing should be developed on Gitter, GitLab, or GitHub or whatev and also be for other sites beyond just SaidIt, in part to diversify and expand and overlap communities, and to take the focus off SaidIt. If Twitter is similar to Reddit in many ways, that means SaidIt is similar to Mastodon and that may be a great resource to help make this happen and to bring into our fold. Or not.

[–][deleted] 2 insightful - 1 fun2 insightful - 0 fun3 insightful - 1 fun -  (3 children)

I love this decentralized archiving idea, but until it exists I can't really put it on the list as a path forward. I would like to work on something cool like that though. You could ensure the data hasn't been tampered with.

[–]JasonCarswell 1 insightful - 1 fun1 insightful - 0 fun2 insightful - 1 fun -  (2 children)

Why not?

It's the far end goal. And it's not that far. It's basically IPFS with extras.

Between here and there are the temporary solutions: Archive.org and Archive.is. Either or both.

Keep a record so when the time comes you can go back and scrape again, maybe from the original and the two archives, keeping in mind it may be different - in which case it might be even more interesting what comparatively changes.

Maybe it's vaporware that will never materialize. Maybe something better will come along. Maybe it will kick around in the back of your mind for years. Maybe one day someone will be glad you brought their attention to it and can make it happen.

Just put the SaidIt decentralized archival backbone last on the list.

[–][deleted] 2 insightful - 2 fun2 insightful - 1 fun3 insightful - 2 fun -  (1 child)

done

[–]JasonCarswell 1 insightful - 2 fun1 insightful - 1 fun2 insightful - 2 fun -  (0 children)

Easy sell.

[–]JasonCarswell 2 insightful - 1 fun2 insightful - 0 fun3 insightful - 1 fun -  (0 children)

Under the full blood red moon the hideous monster raises it's gnarly visage howling into the deathly cold night, "It's all about the Benjamins!"

Maybe, just maybe, there's an opportunity here to 1) openly discuss and transparently develop some ideas about this, and 2) ultimately arrive at a general consensus about ways to not only financially support the archivist posse, assuming that happens, and of course SaidIt. Sustainability.

There are a lot of ways this might happen. Per post or per megabyte or whatever.

I don't even remotely know how it might all work, including some guarantees. Not intentionally, I admit I am not reliable. If I had a computer it may go down for a month or a year until I fix it. If I fix it. Because it's a server in a federation I would feel compelled to participate and get it going. In 5 years I might be weary of it and have other interests. And there may be others who are even worse than I am.

This is a problem. There'd have to be commitments, guarantees, backups, recourses, and redirects. I don't even know where to begin.

One thing I have thought about is shipping/mailing hard drives to quickly transfer vast quantities of data. This would breach anonymity issues for some. But if the ring of archivists were to have their secret Santa on each side not everyone would need to know, though trust would be critical. One drive arrives and you sync it while mailing the same one when your done or there are more than one in play. But you can't just have random stuff on the drive. It has to be organized, and it seems to me the best way to organize it is by date. If we started one for 2019 we'd all be on the same page, kinda like block chain. Maybe we fill a 4tb drive every 2 or 4 or 6 or 12 months. As the drive gets close to full we determine that at midnight of a preset day begins the next dive. Anyone new joining the archive can either slowly torrent the archive from the many or ask and get a cloned drive that they can verify. A further way to shard it, torrents could be time spans or be triggered at a certain megabyte - like blockchain.

I could go on but I should stop. I'm tired and not so clear.

I guess my point is, if "everything I post is worth the indie-archive", then I should help support it, in cash, crypto, home servers, or seed boxes in the cloud.

[–]JasonCarswell 2 insightful - 2 fun2 insightful - 1 fun3 insightful - 2 fun -  (0 children)

2 Archives, 1 Bot = U SaidIt

[–]JasonCarswell 2 insightful - 1 fun2 insightful - 0 fun3 insightful - 1 fun -  (0 children)

Approach #4:

A javascript bookmark type thing in your browser. When on a page you wish to share you click it to run the code. I don't know if that's what is meant in #1 or not. Also the code may 1) activate actions on SaidIt.net, or 2) activate actions on Archive.or, or 3) both.

Approach #5:

All1 incoming new SaidIt posts automatically "ping" the archive.org into action and it spits back the address of the new archived scrape so SaidIt can place an "Archive backup" button somewhere (most likely top right or bottom right of the post).

1 Assuming we all agree, or perhaps an opt-in or opt-out in preferences may be wanted.