<!-- wp:paragraph -->
<p>This guide will walk you through common scenarios which can cause PaperCut to stop working, or the PaperCut Web Server to stop displaying correctly. So... let's get stuck in and see what's going on!</p>
<!-- /wp:paragraph -->

<!-- wp:spacer {"height":20} -->
<div style="height:20px" aria-hidden="true" class="wp-block-spacer"></div>
<!-- /wp:spacer -->

<!-- wp:block {"ref":7017} /-->

<!-- wp:spacer {"height":20} -->
<div style="height:20px" aria-hidden="true" class="wp-block-spacer"></div>
<!-- /wp:spacer -->

<!-- wp:heading {"align":"left","level":3} -->
<h3 class="has-text-align-left">Have there been any recent changes made on the server?&nbsp;</h3>
<!-- /wp:heading -->

<!-- wp:paragraph -->
<p>Often the answer will be a no at first; but if you dig a bit deeper, suddenly the customer remembers they made some changes that they believe are insignificant, but not always are. Some of the changes that the customer may consider not relevant can be things such as:</p>
<!-- /wp:paragraph -->

<!-- wp:list -->
<ul><li>Is it a new Server setup?&nbsp;</li><li>Was this working for a set amount of time, and it stopped working?</li><li>Have there been any recent updates to PaperCut? Even minor updates or patches can cause some issues...</li><li>Has the server recently received any other patches or updates such as OS updates?</li><li>Have there been any updates in the firewall or proxy policies? Often on-site firewall and proxies are managed by external companies, so it is worth it for the customer to double-check with them!</li><li>Do they have a custom SSL certificate installed?&nbsp;</li><li>Have they recently configured PaperCut to force HTTPS? Did they follow <a href="https://www.papercut.com/kb/Main/ForcingSSL">the recommended procedure</a>?</li><li>Has the server (or external database server) recently run out of disk space?&nbsp;</li><li>Is there an issue connecting to the database (if the customer is not using the derby DB but an external DB)?&nbsp;</li></ul>
<!-- /wp:list -->

<!-- wp:heading {"align":"left","level":3} -->
<h3 class="has-text-align-left">Has the service started?&nbsp;</h3>
<!-- /wp:heading -->

<!-- wp:paragraph -->
<p>Are you trying to login to your PaperCut Web Interface and are presented with something similar to this:</p>
<!-- /wp:paragraph -->

<!-- wp:spacer {"height":20} -->
<div style="height:20px" aria-hidden="true" class="wp-block-spacer"></div>
<!-- /wp:spacer -->

<!-- wp:image {"id":8489,"sizeSlug":"large"} -->
<figure class="wp-block-image size-large"><img src="https://staging.selectec.com/wp-content/uploads/2020/03/server-error-1.jpg" alt="" class="wp-image-8489"/></figure>
<!-- /wp:image -->

<!-- wp:spacer {"height":20} -->
<div style="height:20px" aria-hidden="true" class="wp-block-spacer"></div>
<!-- /wp:spacer -->

<!-- wp:paragraph -->
<p>Open services.msc and make sure <a href="https://www.papercut.com/kb/Main/StoppingAndStartingPaperCutServices">PaperCut Application Server service started</a>.</p>
<!-- /wp:paragraph -->

<!-- wp:spacer {"height":50} -->
<div style="height:50px" aria-hidden="true" class="wp-block-spacer"></div>
<!-- /wp:spacer -->

<!-- wp:heading {"align":"left","level":3} -->
<h3 class="has-text-align-left">Is the issue intermittent?</h3>
<!-- /wp:heading -->

<!-- wp:paragraph -->
<p>If the customer is having issues with their devices not connecting promptly or the app service stopping, but then running fine when simply restarting the service, this might point to a resource issue.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>Have a look at the service.log file located at InstallLocation/server/logs/service.log</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>...And you see a lot of entries like this:</p>
<!-- /wp:paragraph -->

<!-- wp:preformatted -->
<pre class="wp-block-preformatted">INFO   | jvm 1    | 2019/01/22 14:58:00 | JVM Process has not received any CPU time for 138 seconds.  Extending timeouts.</pre>
<!-- /wp:preformatted -->

<!-- wp:paragraph -->
<p>^This would indicate a resource issue.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>If the instance is a VM, make sure that you followed the <a href="https://www.papercut.com/kb/Main/PaperCutOnVM">recommendations when running PaperCut on a VM</a>.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>If you are concerned about a resource issue, it is always good to check that nothing else on the server is taking resources away from PaperCut. (Feel free to send us a set of debug logs over a timeframe where an issue occurred and we can run a diagnostic determining if an external application or PaperCut has been taking up the resources on the server.)</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>As a reminder, the handy&nbsp;<a target="_blank" href="https://www.papercut.com/kb/Main/PaperCutServerSizingGuide#server-sizing-table" rel="noreferrer noopener">Server Sizing Guide</a>&nbsp;from PaperCut has a Table that gives you an idea of what resources PaperCut needs - these are guidelines only and may vary depending on the customer.</p>
<!-- /wp:paragraph -->

<!-- wp:spacer {"height":50} -->
<div style="height:50px" aria-hidden="true" class="wp-block-spacer"></div>
<!-- /wp:spacer -->

<!-- wp:heading {"align":"left","level":3} -->
<h3 class="has-text-align-left">You try starting the service, and you see "Error 1069: The service did not start due to a logon failure"...</h3>
<!-- /wp:heading -->

<!-- wp:paragraph -->
<p>This can be because the password is incorrect (don't forget expired or changed passwords can cause this!). To change the password, right-click on the service, choose properties, click the log on tab and have the Admin update the password for this service account.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>This could also happen if the user account is locked out or deleted, and the service has not been updated. If this is the case, simply right click on the service, choose properties, click the log on tab and have the Admin update the account to a user who is still valid and not locked out/deleted.</p>
<!-- /wp:paragraph -->

<!-- wp:spacer {"height":50} -->
<div style="height:50px" aria-hidden="true" class="wp-block-spacer"></div>
<!-- /wp:spacer -->

<!-- wp:heading {"align":"left","level":3} -->
<h3 class="has-text-align-left">You see "Error 1053: The service did not respond to the start or control request in a timely fashion"...&nbsp;</h3>
<!-- /wp:heading -->

<!-- wp:paragraph -->
<p>This happens because PaperCut takes a long time to start up - it's safe to ignore this, so don't panic.&nbsp;</p>
<!-- /wp:paragraph -->

<!-- wp:spacer {"height":50} -->
<div style="height:50px" aria-hidden="true" class="wp-block-spacer"></div>
<!-- /wp:spacer -->

<!-- wp:heading {"align":"left","level":3} -->
<h3 class="has-text-align-left">Is the customer running on an external DB?</h3>
<!-- /wp:heading -->

<!-- wp:paragraph -->
<p>The customer is getting this screen or a similar screen when trying to login to the admin console?</p>
<!-- /wp:paragraph -->

<!-- wp:spacer {"height":20} -->
<div style="height:20px" aria-hidden="true" class="wp-block-spacer"></div>
<!-- /wp:spacer -->

<!-- wp:image {"id":8507,"sizeSlug":"large"} -->
<figure class="wp-block-image size-large"><img src="https://staging.selectec.com/wp-content/uploads/2020/03/system-unavailable-papercut-1-1024x476.png" alt="" class="wp-image-8507"/></figure>
<!-- /wp:image -->

<!-- wp:spacer {"height":20} -->
<div style="height:20px" aria-hidden="true" class="wp-block-spacer"></div>
<!-- /wp:spacer -->

<!-- wp:paragraph -->
<p><strong>Issue:</strong></p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>Problem connecting to the database (in this case, MySQL). This could be any kind of supported external DB which could throw this error.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p><strong>Troubleshooting Steps:</strong></p>
<!-- /wp:paragraph -->

<!-- wp:list {"ordered":true} -->
<ol><li>The first step is to check if the DB service is running by opening services.msc on the DB Server - start up the service if it is not running. Sometimes the Database services are not configured to automatically relaunch on a server reboot, and if you updated the server recently, this could be the reason for this happening.</li><li>If the service is running,&nbsp;<a target="_blank" href="https://www.papercut.com/kb/Main/FirewallPorts" rel="noreferrer noopener">check the ports</a>.&nbsp;</li></ol>
<!-- /wp:list -->

<!-- wp:paragraph -->
<p>If the above does not help, it could also be a mapDB issue (see the section on clearing the mapDB), or the fact that the DB has been put in read-only mode, in which case talk to the DB admin, or change the DB to have the relevant permissions.</p>
<!-- /wp:paragraph -->

<!-- wp:spacer {"height":50} -->
<div style="height:50px" aria-hidden="true" class="wp-block-spacer"></div>
<!-- /wp:spacer -->

<!-- wp:heading {"align":"left","level":3} -->
<h3 class="has-text-align-left">Manually fetch the logs</h3>
<!-- /wp:heading -->

<!-- wp:paragraph -->
<p>If the server is unable to start, the user won't be able to log into the web interface to download the diagnostic bundle.&nbsp;</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>Make a copy of the logs folder, zip this up, and upload it to support (found in: [app-path]/server/logs/). Copying the files to another location first is important, as some ZIP tools will not include files currently in use.</p>
<!-- /wp:paragraph -->

<!-- wp:spacer {"height":50} -->
<div style="height:50px" aria-hidden="true" class="wp-block-spacer"></div>
<!-- /wp:spacer -->

<!-- wp:heading {"align":"left","level":3} -->
<h3 class="has-text-align-left"><strong>Is there an error message like "java.lang.IndexOutOfBoundsException: 0" or "java.io.IOException: Checksum does not match, data broken" or "Error details: java.io.EOFException" or “&nbsp; java.io.IOException: Checksum does not match, data broken at org.mapdb.StoreWAL.get2(StoreWAL.java:336)”</strong></h3>
<!-- /wp:heading -->

<!-- wp:paragraph -->
<p>If you have any of the above error messages, then read on!</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>This can happen when the PaperCut server runs out of disk space or is restarted abruptly, and the mapDB files corrupt. This fix works for some different error messages as well.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>The mapdb is a very small Java database used to keep some things that don't need to be logged into the actual database, but are also not needed in memory so we take the information and store it in those files.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>Sometimes we run into issues where the files can become corrupted and we are unable to read them. Normally, if we cannot fully read the file, it will rebuild itself in the background so no one notices what is happening. On very rare occasions we can read the file but the data isn't valid, so if we see there should be three columns but we only see two or maybe four... it throws us off. Thus, being unable to log in to the PaperCut console.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>Try this:</p>
<!-- /wp:paragraph -->

<!-- wp:list {"ordered":true} -->
<ol><li>Stop the PaperCut Application Server Service</li><li>Move the three files with job event mapDB in the name from PaperCut MF/server/data/internal to the desktop  in older versions of PaperCut the location is under&nbsp;&nbsp;<strong>\PaperCut MF\server\tmp\</strong></li><li>Start the PaperCut Application Server Service</li><li>Wait a minute and attempt to log into the web interface of the server again.&nbsp;</li></ol>
<!-- /wp:list -->

<!-- wp:spacer {"height":50} -->
<div style="height:50px" aria-hidden="true" class="wp-block-spacer"></div>
<!-- /wp:spacer -->

<!-- wp:heading {"align":"left","level":3} -->
<h3 class="has-text-align-left">The error message says "Unable to start server. Address already in use: bind" in the server.log file&nbsp;</h3>
<!-- /wp:heading -->

<!-- wp:paragraph -->
<p>This error suggests that something else is trying to use the same port as the PaperCut server (possibly 9191, 9192, 80, or 443).</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>This can happen for plenty of reasons, but it frequently occurs when customers turn on a Payment Gateway (such as Paypal) which uses port 80 - i.e. there is another service on the operating system that is using this port.  Possible culprits may be IIS (the Windows web server role), BranchCache (peer to peer Windows Update sharing), Sophos and Intel management tools.</p>
<!-- /wp:paragraph -->

<!-- wp:list -->
<ul><li>Open an elevated command prompt Window and run&nbsp;<strong>netstat -aon | findstr :80</strong>&nbsp;to see what processes are also using the port.&nbsp;</li><li>Did it start happening after they enabled a Payment Gateway? To resolve, configure the App Server to use a nonstandard port if possible or remove the conflicting application.</li></ul>
<!-- /wp:list -->

<!-- wp:spacer {"height":50} -->
<div style="height:50px" aria-hidden="true" class="wp-block-spacer"></div>
<!-- /wp:spacer -->

<!-- wp:heading {"align":"left","level":3} -->
<h3 class="has-text-align-left">An error message says "invalid LOC header (bad signature)" in the server.log or service.log file?&nbsp;</h3>
<!-- /wp:heading -->

<!-- wp:paragraph -->
<p>This particular error message suggests that some of the files used by the PaperCut application server may have become corrupted.  This happens when the server was not shut down gracefully; if the disk is full, if the application crashed, or anything remotely like this.&nbsp;</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>You could try removing the files from \PaperCut MF\server\tmp\ or try running the PaperCut application server installer to overwrite those files - this often fixes the error and the associated problems.</p>
<!-- /wp:paragraph -->

<!-- wp:spacer {"height":50} -->
<div style="height:50px" aria-hidden="true" class="wp-block-spacer"></div>
<!-- /wp:spacer -->

<!-- wp:heading {"align":"left","level":3} -->
<h3 class="has-text-align-left">"Unable to start server. Cannot recover key."</h3>
<!-- /wp:heading -->

<!-- wp:paragraph -->
<p>On rare occasions, customers running updates by the book can still come up against issues, leading to the web interface failing to respond after reloading the DB.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>The keystore can corrupt in the transition from the old version to the new version, if this happens, you would need to create a new keystore for the certificate via keystore explorer which will then result in the web interface responding as expected again.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>You can find a guide on how to use Keystore explorer<a target="_blank" href="https://www.papercut.com/kb/Main/SSLWithKeystoreExplorer" rel="noreferrer noopener">&nbsp;here</a>.</p>
<!-- /wp:paragraph -->

<!-- wp:spacer {"height":50} -->
<div style="height:50px" aria-hidden="true" class="wp-block-spacer"></div>
<!-- /wp:spacer -->

<!-- wp:heading {"align":"left","level":3} -->
<h3 class="has-text-align-left">Restoring a corrupt database</h3>
<!-- /wp:heading -->

<!-- wp:paragraph -->
<p>If you are unable to start the PaperCut application service and PaperCut is showing one of the following errors on screen or in the logs:</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p><strong><em>[CODE]</em></strong></p>
<!-- /wp:paragraph -->

<!-- wp:preformatted -->
<pre class="wp-block-preformatted">Failed to start database ‘derby’, see the next exception for details.
The conglomerate (X, Y) requested does not exist.
Recovery failed unexpected problem log record is Not first but transaction is not in transaction table
: X
Container Container(X, Y) cannot be opened; it either has been dropped or does not exist.
Page Page(X,Container(0, Y)) is at version V, the log file contains change version W, either there are log records of this page missing, or this page did not get written out to disk properly.
Invalid checksum on Page Page(X,Container(0, Y))
Container X not found.

</pre>
<!-- /wp:preformatted -->

<!-- wp:paragraph -->
<p>Failed to start database 'derby', see the next exception for details.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>The conglomerate (X, Y) requested does not exist.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>Recovery failed unexpected problem log record is Not first but transaction is not in transaction table</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>: X</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>Container Container(X, Y) cannot be opened; it either has been dropped or does not exist.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>Page Page(X,Container(0, Y)) is at version V, the log file contains change version W, either there are log records of this page missing, or this page did not get written out to disk properly.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>Invalid checksum on Page Page(X,Container(0, Y))</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>Container X not found.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>Your customer most likely has a corrupt database and we would recommend you proceed with a restoration of the database. The instructions for this are&nbsp;<a target="_blank" href="https://www.papercut.com/kb/Main/RestoringACorruptDatabase" rel="noreferrer noopener">here</a>.</p>
<!-- /wp:paragraph -->

<!-- wp:spacer {"height":50} -->
<div style="height:50px" aria-hidden="true" class="wp-block-spacer"></div>
<!-- /wp:spacer -->

<!-- wp:heading {"align":"left","level":3} -->
<h3 class="has-text-align-left">Have you considered Health Monitoring to avoid specific problems?</h3>
<!-- /wp:heading -->

<!-- wp:paragraph -->
<p>Have a look at our blog posts about health monitoring, as setting up health monitoring will help your customers be proactive in spotting potential sticky situations in the future ;-)</p>
<!-- /wp:paragraph -->

<!-- wp:columns -->
<div class="wp-block-columns"><!-- wp:column -->
<div class="wp-block-column"></div>
<!-- /wp:column -->

<!-- wp:column -->
<div class="wp-block-column"></div>
<!-- /wp:column --></div>
<!-- /wp:columns -->

<!-- wp:columns -->
<div class="wp-block-columns"><!-- wp:column -->
<div class="wp-block-column"><!-- wp:core-embed/wordpress {"url":"https://staging.selectec.com/papercut-health-monitoring-grafana-prometheus/grafana/","type":"wp-embed","providerNameSlug":"select-technology-ltd","className":""} -->
<figure class="wp-block-embed-wordpress wp-block-embed is-type-wp-embed is-provider-select-technology-ltd"><div class="wp-block-embed__wrapper">
https://staging.selectec.com/papercut-health-monitoring-grafana-prometheus/grafana/
</div></figure>
<!-- /wp:core-embed/wordpress --></div>
<!-- /wp:column -->

<!-- wp:column -->
<div class="wp-block-column"><!-- wp:core-embed/wordpress {"url":"https://staging.selectec.com/papercut-health-monitoring-grafana-prometheus/","type":"wp-embed","providerNameSlug":"select-technology-ltd","className":""} -->
<figure class="wp-block-embed-wordpress wp-block-embed is-type-wp-embed is-provider-select-technology-ltd"><div class="wp-block-embed__wrapper">
https://staging.selectec.com/papercut-health-monitoring-grafana-prometheus/
</div></figure>
<!-- /wp:core-embed/wordpress --></div>
<!-- /wp:column --></div>
<!-- /wp:columns -->