This entry is written by Curtis, the IncogNET founder and the individual who dealt with this case.
Back in February I saw notification on my phone about a new ticket submitted to our helpdesk titled, “Fix your fucking vuln” – Naturally, I checked this immediately.
A customer had reported to us that when using photorec, a popular data recovery tool, that they were able to recover data on their VPS that they claim was not their own. They did not recognize the recovered data, and assumed that it must have belonged to other IncogNET customers. In fact, the data that they recovered was the private data of their own XMPP users, including files that their own users had shared with one another or had saved, but their unfamiliarity with the tool and how it works led them to believe that this data, somehow, must have been… magically pulled from other servers?
Typically, something like this would not warrant a public response from me. But unfortunately, myself and IncgogNET in general has had a small number of people ask us about it. The user does make some very bold and alarming claims and assumptions, so general concern is of course understandable. As such, I’ve decided to dust off the ol’ IncogNET Blog and start off strong with this post!
Lets dive into it.
The timeline of events is below:
- On Sunday, February 16th at about 3PM: A new ticket, titled, “fix your fucking vuln” was submitted to our helpdesk. Naturally, when the Slack notification came across my phone this caught my eye and I stopped what I was doing to review the matter immediately. They details of their ticket submission is below. The full record and screenshot of our communication is also included in this blog post.
Posted on Sunday 16th February at 15:08
When you use photorec on your VPS, previously deleted files of other users are pulled up. I found stuff with doxing info in it.
You have ignored my email, don’t ignore this now, the vuln could endanger someone. I shouldn’t have to use this shitty form (that loads for a year) to contact you at all, and then jump through all these hoops filling irrelevant data.
Just twenty minutes later, I respond. Between the time the ticket was submitted and our response, I had checked all of our inboxes for any communication received by them. Not a single email was received to any of our inboxes. I suspect that emails from their email provider, a known haven for… questionable users, is auto-rejected by our mail service provider as a means to reduce spam, phishing and other headaches. In any case, our only supported method of contact for items that requires a response from us is via our helpdesk. This person claims they have trouble accessing our portal, yet thousands of users access it monthly without problem. Then again, personal and isolated issues with accessing it are nearly impossible to diagnose and beyond the scope of what we do.
Anyhow, I promptly respond:
Posted on Sunday 16th February at 15:28
Hello,
This is the only notice we’ve received regarding this as far as I can tell. What email address did you contact previously? Nothing in our inbox(es) from [redacted] or containing keywords like “photorec”.
Thank you,
IncogNET
And I never heard back from them.… They never responded, and I shrug it off as someone realizing their mistake. After all, photorec is a very common tool that is used by people everyday to recover lost data on their devices. Reinstall the OS on your laptop, but forgot to backup some wedding photos? Your BitCoin wallet? Some random files? Tools like photorec exist for this purpose, so that you can recover data that is “deleted” on your drive but not yet overwritten by new data. Mistakes and accidents happens, drives fail, data is lost… and there is a slew of products and software aimed at fixing those, “Oh shit!” moments.
Anyhow, a couple months pass since their ticket submission we never hear back. One day, the author decides that the next best course of action, instead of communicating with us, is to create a blog post with some (sadly) comically misleading and very bold claims…
An archived version of this can be found here, though we do not wish to directly link to them. In fact, you can’t even access their blog from IncogNET IP space (Ex: IncogVPN). Either they block all VPN traffic or they’re being petty. They never communicated with us after their initial ticket submission, and we only learned of their blog post through Twitter.
In their blog post they state:
The saga began when my XMPP server was malfunctioning and I couldn’t debug it properly (or just couldn’t be bothered), so I simply reinstalled it. But I forgot to backup account files of some people that were using it, so I tried photorec to recover. I was then quite surprised how my (virtual) drive filled up in seconds. I instantly realized what was up… Curiosity got her way and I downloaded everything that had been lifted and started checking it out, and was shocked. Shocked to find images with doxing info (names), for example. Someone more evil than me might have abused it right then… I couldn’t even be bothered to look through it all but I’m sure I’d be able to find more spicy stuff there if I cared to.
Congratulations. The author just went through the private files of their XMPP server users.
They continue to write:
If you still didn’t figure out what this is about, photorec somehow bypasses Incognet’s virtualization. I have access to many files that I surely did not put there. Executables, sqlite databases (found many on my first try, now only one seems to be recoverable), and who knows what else that a determined attacker might explore for clues. Let me show you an example image I’ve found:
I wish we could take claim for the virtualization stack that we use, as it’s literally used for hundreds of thousands (if not more) Virtual Servers all over the world from a countless number of companies and hosting providers, but alas, we cannot take credit for the KVM Virtualization technology.
And for what it’s worth, I understand the alarm and general concern. I do not blame the author by the original confusion and alarm, assuming it was indeed genuine. If you’re ignorant to how tools like that operate and things like virtualization stacks, their effectiveness may be alarming. But, again, what the author of that blog has discovered was nothing more than the files of his very own XMPP users and old system files from his previous Operating System install.
In any case, I dig up their old ticket that they neglected and reach back out to them.
Posted on Sunday 9th March at 17:54
Hello,
We never heard back from you regarding this, but have seen your Twitter post. We were quite prompt in our response to you here:
You: Posted on Sunday 16th February at 15:08
Our response: Posted on Sunday 16th February at 15:28Twenty minute response time there.
In any case, what you are describing should not be possible. We’re using a very common industry standard software stack to deploy KVM Virtual Servers (https://www.virtualizor.com/), with plans already in motion to migrate to VirtFusion (https://virtfusion.com), for a variety of reasons, namely it’s more stable code base and more trusted development team.
Did you buy this VPS directly from us, or did you get it from someone else? As in, is this a VPS that was once used by another user that gave you access, where you reinstalled the OS?
Please reply to this ticket via the helpdesk, not via email. I’ll add a more prominent disclaimer to automated emails to not reply to them. We don’t pipe responses into our helpdesk from email as they never arrive in the correct format and mess things up, so all matters are expected to be handeled via our helpdesk.
Looking forward to getting to the bottom of this.
Thank you,
IncogNET
Still, no response. No communication. No acknowledgement. We decide to run some tests anyway and respond to the user in more detail.
Posted on Sunday 9th March at 20:51
Hello,
After some review, it appears we can only recover discarded documentation and other files related to common OS components. These aren’t related to anything any other user is doing or has done, it appears they’re simply documentation related files from manpages of system components. For example, when I ran photorec on the same hostnode as your (EU-NL-04) on a the same OS type (Debian 11), it recovered data related to GnuPG.
Here are the steps I took, so you may try as well.
1.) Install testdisk (which includes photorec)
sudo apt install testdisk
2.) Run photorecsudo photorecand select your disk, let it run and do it’s thing.
3.) Create a folder that images will be moved to from the recovered folders.mkdir -p ~/image-recovery && chmod 755 ~/image-recovery
4.) Now, search all of the recup-dir.* folders for any image types, and move them to the folder that was created in step 3.find recup_dir.* -type f \( -iname "*.jpg" -o -iname "*.jpeg" -o -iname "*.png" -o -iname "*.gif" \) -exec mv -i {} ~/image-recovery/ \;
5.) For ease of use, I just used Filezilla to connect to my test server and opened the /image-recovery folder, sorted files by largest to smallest and chose a half dozen or so random ones to view.The files all seemed to be related to documentation of common Debian system components, things like schematics (larger files) or icons (smaller files). I did the same process for text documents and found similar results. Nothing personal, nothing related to any IncogNET customer.
It’s strange though, because in my original test I did download a large random icon image pack, that had a bunch of random .png web icons for web-design. I extracted it, but after reinstalling the OS and running photorec, these items were not discovered. Only the random OS documentation junk as described above.
At this time, seems more like an issue with the OS Templates that are provided by Virtualizor. As mentioned previously, we are already working on a large scale migration moving away from their software for a plethora of reasons, which I guess we can toss this onto the pile of reasons as well. At this time it seems like what is being recovered is the junk removed from the OS template images that Virtualizor creates, whereas with our new VirtFusion control panel it actually just fetches the OS image direct from Debian (Or Ubuntu, or Alma, etc) and uses the official distros.
If the data you’ve recovered differs from this, let us know, though at this time it just seems like junk that Virtualizor likely removed from their OS images to slim down their size or something. It’s still being reviewed from our end but figured I’d reach out. I can create you a new VPS on the same host-node if you want one to test this with so you don’t have to use your production one.
Thank you,
IncogNET
To reiterate, the only files we could “recover” on a fresh operating system install on different VMs on different hypervisors were random documentation files that would typically ship with the full, non-minimal versions of the OS templates. These are not user files, these are just artifacts of of deleted data to slim down the OS image file. It’s also possible that I ramble on and my response was confusing, as they used the above statement as proof that they were… somehow correct.
We submitted one final reply, although we never did hear back:
Posted on Monday 10th March at 12:33
Hello,
To comment on this further, what you have recovered is likely files from your previous OS installation. You mentioned on your blog this was a production server that you reinstalled, and your recovery efforts revealed files that you were unfamiliar with. What you’ve recovered is no different than if you were to reinstall your OS on your computer at home and run similar file recovery tools. You’d likely be able to recover your own data using that method. I’d imagine that what you recovered was files or data that your users had saved or sent to one another that was originally stored on your server.
There is virtually no way that you could recover data or access data from other VMs, only your own. You’re free to research this more. We use KVM Virtualization, which is an industry standard for server isolation. (Less secure virtualization would be like OpenVZ, LXC, etc) Virtual Machine hard disks are a raw storage files located on the host node. For example, your VM’s entire storage is just a 25GB
.rawfile located on the system host node. When you reinstall your server, this file is not recreated , it acts like a normal hard drive for your VM. No other VPS would ever share this same virtual drive, as each VM has it’s own and the operate independently of one another. If your VPS was terminated, as in, fully removed from our system, it’d result in this disk image file being removed as well. A new order would have a new disk image created that matches the VPS ID of that new server and the size of the file would represent the size of the storage space that their VPS plan is supposed to come with.Please research this more and consider updating your blog once you have a better understanding of what you are observing. It’s not a vulnerability, and certainly not files from other IncogNET users you have recovered. The only logical explanation is, that aside from the manpage / documentation bloat that Virtualizor strips from their OS templates that I recovered in my attempt to recreate this is that you are recovering files that was already on your system. We even offer data recovery ISOs in the control panel for customers who need to recover data on their own systems, so this isn’t anything new or unheard of. I believe it’s just an honest misunderstanding.
Happy to assist further if needed.
Thank you,
IncogNET
Still, despite this explanation, our tests, and our rapid response to this accusation the author continues to mislead others in a very disingenuous way. A full log of our communication to one another is shown below for the sake of transparency.
More reasons this is an incredibly stupid claim:
1.) That’s not how Virtual Servers work. You can’t just, somehow, access data and files from other users. Your “hard drive” is a disk image file on the physical hypervisor RAID array. When you reinstall the OS on your VPS, that data is written to the same disk file, just as if you were to reinstall the OS on your laptop or computer.
2.) The software stack that we use is incredibly common place in the web hosting industry. If there was some serious security flaw in the very well maintained and audited KVM Virtualization software, it wouldn’t be discovered by some random guy whose solution to debugging their XMPP server is reinstalling the OS.
3.) If this was, somehow, true, then the web hosting industry in general would have been very severely hit. Many of your favorite hosting providers and big companies would have been impacted and “exploited” overnight. This didn’t happen, of course, because what the author has described is not a vulnerability.
Why are you even responding to this claim?
I’ve been asked to a few times, but honestly? I just couldn’t be bothered to respond to something that I considered baseless and a malicious accusation. It was clear to me that they were not acting in good faith, and between the mention on Twitter and the failed submission to HackerNews about this “possible vulnerability” (Woo, 3 upvotes) with “photorec and VPS hosting providers everywhere”, that they just wanted to promote their blog. In the only direct line of communication they opened with us, I responded to them in 20 minutes. I provided detailed follow ups. The original accusation was, of course, taken seriously and handled professionally from our side. Had this been a serious concern, they’d have made some effort to respond.
In any case, I hadn’t heard about this for months then finally it reappeared on Twitter where we were mentioned, but unable to respond as we are blocked (LOL), but it appears the author deleted their original Tweet anyhow. Cowardly behavior, but it grew apparent to me that they do not want us to respond, that doing so just proves them wrong. So, I think, “What the hell? Let’s do a blog post.” This way, if anyone stumbles across their post we can share our side.
Now, to be completely fair: Since the original “vulnerability” complaint, the author started making actual valid complaints about things like our Amsterdam datacenter migration and ticket response time. Yes, the datacenter migration was quite problematic, lasted far far longer than we had anticipated and many issues arose during this transition period. We sent several emails to impacted customers as well as kept a log on our portal to let customers know the latest updates. We lost a handful of users in this location as a result, and customers since 2021 who have never once reached out to us all of a sudden were wondering what was going on. Heck, we’re not even done going through and applying credit to everyone just yet either. Many many users were impacted, across every product we offer. There will likely be an individual post about this in the future, though you can view the mess of a migration log here. Luckily, all things have been running smoothly since *knock on wood*, though the claims the author did make were quite exaggerated. And yes, our ticket support response time is generally pretty bad right now. We’re a small business, we do not hide this by pretending to be large. We are certainly experiencing growing pains, and we do not make excuses for this. New help will be on-boarded soon to help alleviate this. Additionally, I believe from our response to this “vulnerability” you can see that there is genuine care from our side, that we did respond appropriately.
In closing: What do you think?
Well, what do you think? I’m actually going to leave comments open on this so the topic at hand can be discussed.
All I learned from this is that if I want you to check my ticket faster I should write “fix your fucking vuln”. God bless.