discussion Legacy Attachment System Corruption – Partial Recovery and Archive Release

[Haywoodspartan]

ElasticSearch Deployed and linked to New XenForo

 

[Joquer]

If the hash changed, that usually means the data changed. So how is it possible the attachments still exist, but the files have different bits? Was it a distributed file system? Is XFs hash reliant on the exact metadata (creation date and modification date)?

I have no firsthand knowledge of this specific event, but I do have firsthand knowledge regarding the target insight of this particular question. I have three examples from other related systems and a final SQL example, all, as mentioned, firsthand.
But first I will start with what I understood to be described which is modestly different than the premise you are operating from.. Your premise seems to be that the hash became a new value within the table altogether. That is not what I understood. I interpreted the explanation as the data within the table structure became misaligned- like, say, your Excel Spreadsheet table of your 'Favorite Laughs and the Laughers who Laughed Them' , organized with 6 columns of descriptive categories, plus the basic bio data of the laughers themselves (a linked table, obviously, as a single laugher can have multiple laughs, and I should know), and nearly 800 rows of non-repeating entries, became rendered both mute & moot by your souless Ex, who, having a different opinion on the tembre of a few chuckles and haws, began a willy-nilly delete-fest not of rows nor columns, but of cells., thus shifting part of this column north, of its rightful seat, and these cells left, becoming misaligned to their respective columns, only for this one and that one to shift north yet again..

So you see, if my ID field and your Hash field were linked at the beginning, undoubtedly some of my IDs have new hashes, and some of your hashes have no IDs.

Another way in which I have witnessed this which is possibly more accurate, at least in some ways, to this event, is with music on a usb drive that plugged innto a car stereo I had once many moons ago. This was maybe welll, that part's not relevant, this part is: I would be listening to some B of H from the drive when I turned the car of off. When I turned it on it would start playing some DP. Turn it off while listening to NC, turn it on to JC. Then one day I'm listing to DPwhenallofthesuddenitsnotDPbutNC. Like someone had spliced one tune directly into another. By the time I caught on nearly ever tune on the drive had some form of corruption. I did not invest a great deal of energy into the cause, but evidence suggested it was related to how the system 'spun up' and the folder depth of the last played tune as there was some consistency in the track numbers of the spliced tunes, but pnly the barest pattern in the artists spliced.

But look at the time! I'm afraid I've exhausted my example quota for the day... Hopefully this helped. And if it didn't help, hopefully it didn't hurt. And if did neither, please check your pulse!
Sol Long, and Star Wide!
[automerge]1772866270[/automerge]
Realizing I am prone to the TLDR-tag, I wanted to draw attention to this comment/request/idea I pitched previously:

I, like many have here I'm sure, possess a couple, 4, or more thousand gigs of site-worthy content dating from yesterday back to the midteens or so.

Is there a donation station planned? Or perhaps a better, easier to manage option- those of us so willing and able inventory our goodies, gawdies, and lawdies-help-me-jeebus! in a format easily interpreted and accessible to All Ye Lewd Powers That Be, and you let us know, ahem... When & Where to Stick It!

I am happy to help in any way I can.

 

[Jack Of Blades]

I have no firsthand knowledge of this specific event, but I do have firsthand knowledge regarding the target insight of this particular question. I have three examples from other related systems and a final SQL example, all, as mentioned, firsthand.
But first I will start with what I understood to be described which is modestly different than the premise you are operating from.. Your premise seems to be that the hash became a new value within the table altogether. That is not what I understood. I interpreted the explanation as the data within the table structure became misaligned- like, say, your Excel Spreadsheet table of your 'Favorite Laughs and the Laughers who Laughed Them' , organized with 6 columns of descriptive categories, plus the basic bio data of the laughers themselves (a linked table, obviously, as a single laugher can have multiple laughs, and I should know), and nearly 800 rows of non-repeating entries, became rendered both mute & moot by your souless Ex, who, having a different opinion on the tembre of a few chuckles and haws, began a willy-nilly delete-fest not of rows nor columns, but of cells., thus shifting part of this column north, of its rightful seat, and these cells left, becoming misaligned to their respective columns, only for this one and that one to shift north yet again..

So you see, if my ID field and your Hash field were linked at the beginning, undoubtedly some of my IDs have new hashes, and some of your hashes have no IDs.

Another way in which I have witnessed this which is possibly more accurate, at least in some ways, to this event, is with music on a usb drive that plugged innto a car stereo I had once many moons ago. This was maybe welll, that part's not relevant, this part is: I would be listening to some B of H from the drive when I turned the car of off. When I turned it on it would start playing some DP. Turn it off while listening to NC, turn it on to JC. Then one day I'm listing to DPwhenallofthesuddenitsnotDPbutNC. Like someone had spliced one tune directly into another. By the time I caught on nearly ever tune on the drive had some form of corruption. I did not invest a great deal of energy into the cause, but evidence suggested it was related to how the system 'spun up' and the folder depth of the last played tune as there was some consistency in the track numbers of the spliced tunes, but pnly the barest pattern in the artists spliced.

But look at the time! I'm afraid I've exhausted my example quota for the day... Hopefully this helped. And if it didn't help, hopefully it didn't hurt. And if did neither, please check your pulse!
Sol Long, and Star Wide!
[automerge]1772866270[/automerge]
Realizing I am prone to the TLDR-tag, I wanted to draw attention to this comment/request/idea I pitched previously:

I, like many have here I'm sure, possess a couple, 4, or more thousand gigs of site-worthy content dating from yesterday back to the midteens or so.

Is there a donation station planned? Or perhaps a better, easier to manage option- those of us so willing and able inventory our goodies, gawdies, and lawdies-help-me-jeebus! in a format easily interpreted and accessible to All Ye Lewd Powers That Be, and you let us know, ahem... When & Where to Stick It!

I am happy to help in any way I can.

what do you mean by (Is there a donation station planned? Or perhaps a better, easier to manage option- those of us so willing and able inventory our goodies, gawdies, and lawdies-help-me-jeebus! in a format easily interpreted and accessible to All Ye Lewd Powers That Be, and you let us know, ahem... When & Where to Stick It!)

 

[Perverticus]

will i have to regrind m+ and p5? or make a new account?

 

[boserwolf]

You must be registered to see attachments

i am also working on lewdcord, the alternative to discord.

Noice
The "LC" Discord isn't really a "LC" Discord. (No M+ stuff)

 

[Jack Of Blades]

will i have to regrind m+ and p5? or make a new account?

nope

 

[misterfluffums]

Will you vet the image data before you upload the archive?

If the hash doesn't match the database, it's likely the images are corrupted and can't be viewed. Doesn't really have a point to upload a bunch of corrupted images.

And maybe more important: is there a way to verify you only upload images from publicly available attachments? Otherwise there's a chance that members who shared pictures in DMs get accidentally doxxed by EXIF metadata in pictures they never expected to be made public.

I would not rush to publish this archive but really think about it for at least a week.

 

[GoonerAlien]

this is like when you first install linux and fuck it all up, not a jab at you guys, but I complete understand.
the amount of times I had to reinstall Linux is crazy, when I was younger coming from Microsoft, 10 years ago.
At the end of the day, error is key to success.

 

[SilentHunter]

All that remains is to wish good luck and success to the staff and administrators. The main thing is to keep links to the games. About other things (videos, pictures, comics), I personally do not regret, I came here for the sake of games. Bank? If it gets lost during all the current perturbations, then it's not a pity either, although the interest collection game was quite fun. And I want to thank the staff for their work - the last three months have been a very difficult period.

 

[letmeinthepub]

Will you vet the image data before you upload the archive?

If the hash doesn't match the database, it's likely the images are corrupted and can't be viewed. Doesn't really have a point to upload a bunch of corrupted images.

And maybe more important: is there a way to verify you only upload images from publicly available attachments? Otherwise there's a chance that members who shared pictures in DMs get accidentally doxxed by EXIF metadata in pictures they never expected to be made public.

I would not rush to publish this archive but really think about it for at least a week.

As far as I know, all attachments had EXIF metadata striped on upload.

 

[CHEGOBNK]

LewdCorner is dead, long life LewdCorner 2!
It's a shame that new users (and me) who get to higher ranks will never see rank-restricted threads from the past. But welp, nothing can be done about that.

 

[GoonerAlien]

and im not trying to be a dick whatsoever, I honestly only check in every so few months
whatever happened to Heisenberg? Did he give the site away due to his mental health, or was Heisenberg never a real person. I am honestly so confused, but I hope all is all.

 

[Rational Chaos]

The best solution at present is to work on a new site, save what can be saved and allow users to download data archives. In this way, over time, the data archives can be restored to the new server, because at some point the Phoenix must rise from its ashes.

But at this point, I wonder: will what is published now be saved on the new server?

 

[misterfluffums]

and im not trying to be a dick whatsoever, I honestly only check in every so few months
whatever happened to Heisenberg? Did he give the site away due to his mental health, or was Heisenberg never a real person. I am honestly so confused, but I hope all is all.

He changed his username, jackofblades = heisenberg

 

[Jack Of Blades]

He changed his username, jackofblades = heisenberg

and im not trying to be a dick whatsoever, I honestly only check in every so few months
whatever happened to Heisenberg? Did he give the site away due to his mental health, or was Heisenberg never a real person. I am honestly so confused, but I hope all is all.

You must be registered to see attachments

I change my username yearly lol.

 

[Mr.LewdRim]

Ah well shit happens time to move forward. ETA om the shop returning lol.

 

[hardhead]

well, least we know better now. something is certainly better then nothing right boys?
[automerge]1772877171[/automerge]

You must be registered to see attachments

I change my username yearly lol.

love that response lol. also satan to sir satan to crowley, i noticed a trend i think.

 

[Noci]

So, the fumo thread is unrecoverable now? shit. i guess i can manually re-upload all the pictures, but it wouldn't be the same now, would it?
:_(

 

[VSCDB]

So you see, if my ID field and your Hash field were linked at the beginning, undoubtedly some of my IDs have new hashes, and some of your hashes have no IDs.

I see where you are going, but that's not really how database formats usually work, to be able to open the db, and being able to still read it. In 99% of cases where an SQL database gets messed up by actual corruption and not the fuck up of the one writing the SQL statement, you will have either full corruption, like full pages being corrupted, data being partially written or truncated, or the DB format being inconsistent, and before getting actually corrupt data you will likely get checksum mismatches and index errors. To have the bits of the binary realign itself that perfectly, is extremely unlikely, possible, but I would say it's probably way more likely that I get hit by lightning in the next 10 minutes, while sitting in my room in the sun. It is much more likely that the whole format file is messed up, not just the rows having switched ids or hashes, and you can't read much data fully anymore. So in every normal case the underlying hashed data changed (the attachments), or settings changed and a different hash is now used. Like I mean it, the chances for this happening with any modern engine are almost zero, even more so if they used common filesystems, since almost all of them split up their files physically on the storage in some way.
They can have index corruption, like secondary index corruption behaving like switched data, but the file bits will not randomly have changed, and these cases you can rebuild the index, and in primary index corruption you would in most cases not even get much data back anymore anyway, since where the data lives is not actually known reliably anymore. For this to happen it would require page level corruption (modern DBs usually store their data in pages with a fixed size), where all row and field metadata (headers, offsets, pointers and indices) are still valid and untouched, but the actual data bits and bytes are swapped, this is literally like winning the Roulette table 4 times in a row while betting on 2.

 

[sirneversleep]

Meh, I don't really care about whatever hidden section y'all are using that will certainly not lead someone to jail one day, I just want a functioning website to access games not permitted by the other website that I won't name because whoever coded this website felt too immature not to hijack whenever it's typed. Oh and I'd like maybe for one fucking day not always having to be force fed whatever drama the admin has that NEED everyone to read. I just want the thumbnail in the latest updates page to work. That's all.

Agreed. I don't need the daily drama updates. I just want the latest updates page. That is literally all I care about and the only reason I'm here for. (Aside from ATF shutting down, but I was here long before that. That's just adding another reason.)