Take a look at /etc/security/time.conf
To limit for example ssh access from 23:00 PM and 08:00 AM.
The format of the file is:
the !Al means, anything except "All the days".
If you would like to permit people from 4 to 8 PM all the days, except root:
Further reading: man time.conf