Category: Troubleshooting

Issues installing Netbox web service

Nils Magnus Eliassen3 Comments

Hi all

Today I was trying to install netbox on a Linux server. I have done it before but just for testing and never connected it to apache or nginx for a production enviorment. I used the guide supplied in the official manual but I managed for some reason to fail completely. Not sure where the error was if I just did something completely stupid orthe manual was just incomplete. You can check the manual here if you need.

My issue was that when all the commands I had to run for the HTTPS server according to the manual I still got an error. The lovely 503 Service unavailable.

Service Unavailable
The server is temporarily unable to service your request due to maintenance downtime or capacity problems. Please try again later.

Apache/2.4.41 (Ubuntu) Server at netbox.example.com Port 443

When I checked the service I got the following response:

manager@ubuntults:~$ systemctl status netbox
● netbox.service - NetBox WSGI Service
     Loaded: loaded (/etc/systemd/system/netbox.service; enabled; vendor preset>
     Active: activating (auto-restart) (Result: exit-code) since Tue 2020-05-26>
       Docs: https://netbox.readthedocs.io/en/stable/
    Process: 1217 ExecStart=/usr/local/bin/gunico --pid /var/tmp/netbox.pid --p>
   Main PID: 1217 (code=exited, status=203/EXEC)
lines 1-6/6 (END)

In the active section you can see that it says activating and not activated and Main PID: gives me: (code=exited, status=203/EXEC)

This tells me that the system has issues executing the files for starting the service. I followed this up by going to the service file where all the settings are made for the service.

sudo nano /etc/systemd/system/netbox.service

I then continued to look at the execution parameters for the service.

ExecStart=/opt/netbox/venv/bin/gunicorn --pid /var/tmp/netbox.pid --pythonpath /opt/netbox/netbox --config /opt/netbox/gunicorn.py netbox.wsgi

When I checked for the gunicorn file in the filesystem I couldn’t find the file gunicorn in /opt/netbox/venv/bin/. Therefor I entered the command “which gunicorn” that returns the correct path for gunicorn to me.

manager@ubuntults:~$ which gunicorn
/usr/local/bin/gunicorn

The correct syntax would then be:

ExecStart=/usr/local/bin/gunicorn --pid /var/tmp/netbox.pid --pythonpath /opt/netbox/netbox --config /opt/netbox/gunicorn.py netbox.wsgi

When you have done the change you need to update the service like I have done below:

manager@ubuntults:~$ sudo nano /etc/systemd/system/netbox.service
[sudo] password for manager:
manager@ubuntults:~$ sudo nano /etc/systemd/system/netbox.service
manager@ubuntults:~$ systemctl daemon-reload
==== AUTHENTICATING FOR org.freedesktop.systemd1.reload-daemon ===
Authentication is required to reload the systemd state.
Authenticating as: manager
Password:
==== AUTHENTICATION COMPLETE ===
manager@ubuntults:~$ systemctl restart netbox
==== AUTHENTICATING FOR org.freedesktop.systemd1.manage-units ===
Authentication is required to restart 'netbox.service'.
Authenticating as: manager
Password:
==== AUTHENTICATION COMPLETE ===
manager@ubuntults:~$

Now the server should work perfectly 🙂

I couldn’t have solved it hadn’t it been for this post and the answer from Russell Molimock. That put me on the correct track for finding the issue 🙂

Advertisement

Can’t upload image to wordpress due to large size

Nils Magnus Eliassen7 Comments

Hi

In my new job I have been working a little bit with cPanel and WordPress. It sounds easy enough and both of the products are easy enough normally. But there is always one of these fun errors that just makes you wonder. What the f*** could be the problem here. Everything seems fine but everything is so wrong.

When the users where going to upload files we changed the upload limit on the PHP server and everything should be working fine. The users should be able to upload files up to 250 MB. At least that was what I was thinking. But no matter what I did I got the following error up: “Post-processing of the image failed. If this is a photo or a large image, please scale it down to 2500 pixels and upload it again”

My picture was below this size and I belive I tried every single solution to this issue I could find on the internet. The only solution that worked for me was the plugin Add from Server, with this plugin I was able to get my pictures to the libary.

After hours of googling and talking to coworkers we suddenly discovered that cPanel is not delivered with mod_fcgid turned on in Apache. This simple feature took me ages to find out. In addition you need to install the imagick module.

imagick you can install in cPanel by go to Software -> Module Installer in the left menu. Manage the PHP PECL and search and install imagick for all needed PHP versions (at least where you run WordPress).

Then you need to enable mod_fcgid. This can be done in Software -> Easy Apache 4 on the left side menu and choose Customize on the menu showing. Now you can go to Apache modules and search for mod_fcgid. When this module is enabled you can try to upload again. Hopefully it works now!

Be aware that the changes done here needs to be done by your hosting provider and unless you have access to the WHM console you need to forward them this article to fix the issues.

Centos 7 with cPanel loose network config

Nils Magnus EliassenLeave a comment

Today I have been struggeling with cPanel. I did install an Centos 7 server with cPanel WHM on it. I followed the instructions from tecmint and it was pretty straight forward! I have to admit, that was my thought all the way until I rebooted the server with an software update.

At firdst I didn’t know what happened. I thought it was a firewall issue etc. But I couldn’t find any issues. When I finally started to check the server more closely I discovereed when running ifconfig that there where no network card configured with the correct IP. I then moved further and tried to restarting the network service with the command: 

systemctl restart network.service

When doing this I got an error message :
Job for network.service failed because the control process exited with error code. See “systemctl status network.service” and “journalctl -xe” for details.

Checking “systemctl status network.service” gave me the following error message:

Failed to start LSB: Bring up/down networking.

This led me out on a desperate google search that lasted for a couple of hours. I found alot of articles that could be helpful but none that helped me. Or I did, in the end but it still felt like forever. I had read something similear to the solution below on my google searches. But those articles only said I needed to touch the network file (just create an empty file). But according to the forum post below I needed to add the commands below to /etc/sysconfig/network file.

NETWORKING=yes
GATEWAY=XXX.XXX.XXX.XXX

XXX.XXX.XXX.XXX represent the IP of the gateway for the CentOS server. When the file is edited I did a reboot and it all worked again! 🙂 I tried to restart the network.service but for some reason I had to do a complete reboot for it to work.

Forum post

Management access list on Juniper SRX

Nils Magnus EliassenLeave a comment

Sometimes you need to restrict access to your Juniper SRX firewall, or lets say you should always restrict the access to the firewall when it’s connected to the internet.

So now I’ll show you how to limit the traffic to your firewall to only be accessible from the allowed IP’s or IP ranges. In the first part we will insert the subnets and IP’s into a prefix list called manager.

set policy-options prefix-list manager-ip 192.168.4.254/32
set policy-options prefix-list manager-ip 10.0.0.0/8

The next part will tell the firewall to discard requests done to the firewall except to the IP’s in the prefix list above named manager. The filter will only apply to the services below so make sure to not remove any of the services you wan’t to limit access to. The last line in the snippet below is a block command to block off all the traffic hitting this term.

set firewall filter manager-ip term block_non_manager from source-address 0.0.0.0/0
set firewall filter manager-ip term block_non_manager from source-prefix-list manager-ip except
set firewall filter manager-ip term block_non_manager from protocol tcp
set firewall filter manager-ip term block_non_manager from destination-port ssh
set firewall filter manager-ip term block_non_manager from destination-port https
set firewall filter manager-ip term block_non_manager from destination-port telnet
set firewall filter manager-ip term block_non_manager from destination-port http
set firewall filter manager-ip term block_non_manager then discard

In the end we will accept everything to allow everything and by that only allowing access to the IP’s mentioned since they where excepted from the prevoius term that was blocked in the end.

set firewall filter manager-ip term accept_everything_else then accept

At the very end we add the filter to the loopback interface. By doing this all traffic will hit the rule, you can also add it to seperate interfaces if you only want to limit the traffic from certain interfaces.

set interfaces lo0 unit 0 family inet filter input manager-ip

When you have done that the firewall is a bit more secure than before. Hopefully I will have the chance to write how to secure your Juniper SRX even more later on. Below is all the code in one snippet for easy cut and paste.

set policy-options prefix-list manager-ip 192.168.4.254/32
set policy-options prefix-list manager-ip 10.0.0.0/8
set firewall filter manager-ip term block_non_manager from source-address 0.0.0.0/0
set firewall filter manager-ip term block_non_manager from source-prefix-list manager-ip except
set firewall filter manager-ip term block_non_manager from protocol tcp
set firewall filter manager-ip term block_non_manager from destination-port ssh
set firewall filter manager-ip term block_non_manager from destination-port https
set firewall filter manager-ip term block_non_manager from destination-port telnet
set firewall filter manager-ip term block_non_manager from destination-port http
set firewall filter manager-ip term block_non_manager then discard
set firewall filter manager-ip term accept_everything_else then accept
set interfaces lo0 unit 0 family inet filter input manager-ip

Source:
https://www.juniper.net/documentation/en_US/junos/topics/example/permitted-ip-configuring.html

Limit device traffic to only one MX uplink

Nils Magnus Eliassen1 Comment

Hi all

The Meraki MX devices gives you an easy way of automaticly use 2 uplinks. It works seamlessly but it’s hard to do some configuration that is possible on other Cisco devices.

One of those is to deny specific devices to connect over only 1 of the uplinks. Let’s say that WAN 1 is a fiber connection. You got enogh capacity to send and receive all kind of traffic. WAN 2 on the other hand is a sattelite connection. The 2 big drawbacks with sattelite is latency and speed. Sometimes even the cost per MB transferred. Often the guaranteed bandwith on a satelite connection could be as low as 64 kb/s. It’s not much bandwith for other devices then.
wanmeraki

Then the big question is, how do you limit the connection to only use WAN 1. This could be a device that sync data every hour and would generate traffic or a whole subnet with guest wifi users. You don’t want these devices use your costly satelite connection. You do most likely need it for business critical applications.

I spent a few good hours trying to find a solution to this. I asked help from Meraki and various forums. I always got told that traffic shaping should help etc. But the only thing it does is giving me the preferred uplink, it never blocks the traffic from going online if the other WAN connection is down.

The solution I came up with was to turn off NAT when you use the interface that should be blocked. All the devices behind the Meraki that should be blocked does need to be in a seperate VLAN. In version 15 you can exclude specific VLAN from the NAT policy on the uplinks. The traffic will then stop since there is no return route for the traffic (as long as youo don’t add a static route).

So in the above example we want VLAN 20 to only have access over WAN 1 and not WAN 2. You start by finding the network you want to do the change on in the Meraki Dashboard. Then go to Security & SD-WAN and Adressing and VLAN’s on the left side. In the bottom of the page you have NAT exceptions where you can choose to disable NAT on the different uplinks. In my screenshot I have excepted Crew network from the NAT policy. With this config the devices on Crew VLAN can’t use WAN 2/Uplink 2.
NAT meraki

Backup and restore config of Mobility Express.

Nils Magnus EliassenLeave a comment

Hi all

Lately I have been working with the mobility express AP’s from Cisco.  One of the important things to do when you set up new equipment is to have a backup and restore policy for the config.. I chose the easy way out using tftp, it’s the quickest and easiest way to transfer files as long as you have the tftp server secured. The other option you have is ftp.

transfer upload mode tftp
Sets the mode to tftp, you can also choose ftp but then you need to add in username and password too.

transfer upload datatype config
Choose config as the information to store on the server

transfer encrypt enable
Turns on encryption for the file

transfer encrypt set-key supersecret
Gives the encryption a password

transfer upload serverip 10.10.10.10
Gives the ME an IP to the server where to store the config

transfer upload filename MEconfig.cfg
Filename for the config.

transfer upload start
Start the upload.

transfer upload mode tftp
transfer upload datatype config
transfer encrypt enable
transfer encrypt set-key supersecret
transfer upload serverip 10.10.10.10
transfer upload filename MEconfig.cfg
transfer upload start

You should then get the following output.

Mode……………………………………… TFTP
TFTP Server IP…………………………….. 10.10.10.10
TFTP Path………………………………….
TFTP Filename……………………………… MEconfig.cfg
Data Type…………………………………. Config File
Encryption………………………………… Enabled

Are you sure you want to start? (y/N) y

File transfer operation completed successfully.

So far you have done the backup. Then the second most important thing comes, do the restore. It’s almost the same, but you swap out upload with download.

transfer download datatype config
transfer download mode tftp
transfer encrypt enable
transfer encrypt set-key supersecret
transfer download serverip 10.10.10.10
transfer download filename MEconfig.cfg
transfer download start

After the commands have been entered you should see the following output.

Mode............................................. TFTP
Data Type........................................ Config
TFTP Server IP................................... 10.10.10.10
TFTP Packet Timeout.............................. 6
TFTP Max Retries................................. 10
TFTP Path........................................
TFTP Filename.................................... MEconfig.cfg
Encrypt/Decrypt Flag............................. Enabled

Warning: Downloading configuration will cause the controller to reset...

This may take some time.
Are you sure you want to start? (y/N) y

TFTP Config transfer starting.

TFTP receive complete... updating configuration.

CCO Username & Password will NOT be imported. Please Re-Configure the Credentials 'transfer download ap-images cco-username '
'transfer download ap-images cco-password ' after bootup for Image Download

TFTP receive complete... storing in flash.

Sync config to peers.

System being reset.

 

Cisco and python programming

Nils Magnus EliassenLeave a comment

I have decided I want to try to program cisco switches and devices using python. At the moment my programming skills are limited to simple if and else. I now have a plan to configure a linux server with python on to do all my scripting.

The plan is to try to upload the different scripts I create here as I go along. Hopefully it will get useful for others too in the end and not just something for me. I will try to use various youtube videos and pages to learn me the diffferent things with oython and will link them from my upcoming posts as I go along.

If you have something you want me to write about or create a script it’s greatly apreciated!

Problems with NSM after schema upgrade.

Nils Magnus EliassenLeave a comment

The other day we upgraded the schema on our NSM server from 327 to 329. After the upgrade the devices was not able to connect to our NSM anymore. In the deviceDeamon I got the following error:

[Notice] [3078149840-connectionMgr.c:2329] SSH Protocol is not enabled -- DeviceBroker is not ready for incoming device connection.
[Notice] [3078149840-connectionMgr.c:2318] Incoming TCP connection from SSH, device ip x.x.x.x
[Notice] [3078149840-connectionMgr.c:2329] SSH Protocol is not enabled -- DeviceBroker is not ready for incoming device connection.
[Notice] [3078149840-connectionMgr.c:2318] Incoming TCP connection from SSH, device ip x.x.x.x
[Notice] [3078149840-connectionMgr.c:2329] SSH Protocol is not enabled -- DeviceBroker is not ready for incoming device connection.
[Notice] [3078149840-connectionMgr.c:2318] Incoming TCP connection from SSH, device ip x.x.x.x
[Notice] [3078149840-connectionMgr.c:2329] SSH Protocol is not enabled -- DeviceBroker is not ready for incoming device connection.
[Notice] [3078149840-connectionMgr.c:2318] Incoming TCP connection from SSH, device ip x.x.x.x
[Notice] [3078149840-connectionMgr.c:2329] SSH Protocol is not enabled -- DeviceBroker is not ready for incoming device connection.

I didn’t know that a simple schema upgrade could do something to the NSM that would not allow the devices to connect so I ended up contacting JTAC support. When I got a support engineer and explained him the issue he found another error message in the guiDaemon. The error was “DC not connected”.

After a while with troubleshooting the engineer discovered that the issue was the RSA key that is responsible for the communication between the NSM services  and the guiDaemon and devDaemon. The engineer then navigated to devSvr.cfg under /usr/netscreen/DevSvr/var and deleted the RSA keys (ourRsaPrivateKey and theirRsaPublicKey).

After that all the devices in some magical way connected again!

Secondary node locked when commiting

Nils Magnus EliassenLeave a comment

The other day I got a problem with one of my SRX clusters when I was running a commit. The commit was not able to complete and I got the following error:

{primary:node0}[edit]
srx1400# commit
node1:
error: configuration database modified
node0:
error: remote lock-configuration failed on node1

The reason for this error is some uncommited configuration on the secondary node. Earlier the same day I changed the primary for redundancy-group 0 and I guess that I didn’t commit all the config on node1 before changing to node0.

To solve this I had to go into the secondary node (node1) and rollback the uncommitted configuration. Normally you can use OOB to connect to the secondary node but I dont have it at this location. So I have to connect to the secondary node trough the primary node. This is done with the following command on branch devices (SRX650 and below):  request routing-engine login node 1
On High end devices like the one I’m working on (SRX1400 and above) you use: rlogin -T node1

{secondary:node1}% rlogin -T node1
root@srx1400>
--- JUNOS 11.4R9.4 built 2013-08-22 06:24:21 UTC
{secondary:node1}
root@srx1400> configure
warning: Clustering enabled; using private edit
error: shared configuration database modified

Please temporarily use 'configure shared' to commit
outstanding changes in the shared database, exit,
and return to configuration mode using 'configure'

As you can see from the error I have to use configure shared to be able to edit the configuration.

root@srx1400> configure shared
Entering configuration mode
The configuration has been changed but not committed

Before entering the rollback command you can check the uncommitted configuration by running show | compare. This will display all the uncommited configuration

{secondary:node1}[edit]
root@srx1400# show | compare
[edit access profile unos clientjunos]
- pap-password "$9$2V4GDikP5T3fTrvLXwsz36C0B"; ## SECRET-DATA
+ pap-password "$9$jhHP5QF/CA09AxdsYGUp0BRyl"; ## SECRET-DATA

Now you can rollback the uncommited config, check that there is any uncommited config left and exit the configuration mode.

{secondary:node1}[edit]
root@rx1400# rollback
load complete

{secondary:node1}[edit]
root@srx1400# show | compare

{secondary:node1}[edit]
root@srx1400# exit
Exiting configuration mode

{secondary:node1}
root@srx1400>

Now you can close the session and try to commit the configuration from the primary node again. It worked for me! 🙂

As a note I also know that alot of people has had a success of using just the command commit synchronize force on the primary node but it does not work for everyone.

dst-port error in NSM

Nils Magnus EliassenLeave a comment

I got a new error while updating a SRX650 from the Juniper Network and Security Manager. The error started after I upgraded the SRX650 to 12.1X47-D30. The error I got is shown below:

Error Code: 

Error Text:
   Update fails UpdateDevice Results
sanityCheckCmd Success.
lock Success.
GenerateEditConfig Failed .
<?xml version="1.0" encoding="UTF-8"?>
<rpc-reply message-id="1" xmlns="urn:ietf:params:xml:ns:netconf:base:1.0" xmlns:junos="http://xml.juniper.net/junos/12.1X47/junos" xmlns:nc="urn:ietf:params:xml:ns:netconf:base:1.0">
   <rpc-error>
      <error-severity>error</error-severity>
      <error-info>
         <bad-element>dst-port</bad-element>
      </error-info>
      <error-message>syntax error</error-message>
   </rpc-error>
   <rpc-error>
      <error-severity>error</error-severity>
      <error-info>
         <bad-element>dst-port</bad-element>
      </error-info>
      <error-message>syntax error</error-message>
   </rpc-error>
</rpc-reply>


unlock  Success .


Error Details:
   

Logs:
<configuration>
  <version>12.1X47-D25.4</version>
  <system>
    <host-name>casur-srx650-cluster</host-name>
  </system>
  <security>
    <nat>
      <destination>
        <rule-set>
          <name>ca-camera</name>
          <rule>
            <name>camera-01-8200</name>
            <dest-nat-rule-match>
              <destination-port operation="delete">
                <name>8200</name>
              </destination-port>
              <destination-port operation="create">
                <dst-port>8200</dst-port>
              </destination-port>
            </dest-nat-rule-match>
          </rule>
        </rule-set>
      </destination>
    </nat>
  </security>
</configuration>

It’s saying that the destination nat section has problems setting the dst-port. For some reason it was deleting the value and creating it with a new command (dst-port).

I then checked the supported Junos versions on the NSM and I discovered that the last supported version was 12.1X47-D25. Did the downgrade and updated the OS in the NSM. Still the same error as before.

Spoke to JTAC and they informed me that this error was known and that it would help downgrading to D15. This was due to a changed command in Junos. I downgraded to D15 but still the same issue. Researched a bit myself and discovered that it was introduced between X46 and X47.

Earlier it had not been possible to downgrade the versions in NSM. But for some reason I was able to do it now. First from D30 to D25, and after that from X47D15 to X46D40. When I reached  X46D40 I was able to run the update and everything was working.