Forgive me if this was asked before, the results form a search with the key word “/etc/hosts” are quite huge. I frequently encounter dns trouble and would like to populate /etc/hosts with correct routing info. I can’t always change dns servers because the WiFi is set up to restrict dns access sometimes. I don’t care how big it gets, several GIBs is fine I’m just really sick of the message “Could not resolve host name” in fireofox, curl, wget, etc. If you could point out some way to filter what gets in /etc/hosts that would be even better.
It is not quite clear to me what you want. /etc/hosts does not contain routing information. It contains a host/domainnames vs. IP addresses table. A sort of telephone directory. As having all IP addresses and all host/domainnames existing in the world in every /etc/hosts on all systems in the world will grow that file into an unmanageble mass, they invented DNS, where al that data is in a world wide distributed hierarchical database.
/etc/hosts can still be used to have entries in there. Often they are entries for a local LAN (or a bit wider) that are not to be found in DNS (when I want to give my system, printer and router names, I can then refrain from requiring a domain name and setting up a DNS server for that domain). Of course others on the internet wil never know my names andd IP addresses, but they won’t care I assume.
So you can indeed put all the host/domainnames and their IP addresses in your local /etc/hosts. Theoretical. But how would you think you get that information and add it to /etc/hosts? A few can be added of course through an editor (or YaST), but not thousands. At least when you need time to use your system for other things.
So, at those times you have access to a DNS server you could do:
henk@boven:~> nslookup www.google.com
Server: 194.109.6.66
Address: 194.109.6.66#53
Non-authoritative answer:
Name: www.google.com
Address: 173.194.65.105
Name: www.google.com
Address: 173.194.65.99
Name: www.google.com
Address: 173.194.65.104
Name: www.google.com
Address: 173.194.65.106
Name: www.google.com
Address: 173.194.65.103
Name: www.google.com
Address: 173.194.65.147
henk@boven:~>
and then enter all those in your /etc/hosts. You could try to write a script that adds these addresses to it. But it still is a work of Sisyphus. And even then, the first www.google.com page you load with your browser will probably redirect to another host and also have myriad links to other hosts. You first must have all of these in your /etc/hosts before you are done.
And of course, tomorrow one of these IP addresses changes and that will be automalticaly propagated through DNS, bit not to your /etc/hosts. /etc/hosts will grow out-of-date at a remarlable speed.
Your problem is quite common, but your solution isn’t.
Instead of populating /etc/hosts which would be a very manually intensive endeavor,
The standard solution to your problem is to configure your network configuration to point to reliable DNS servers on the Internet.
If you google “tier 1 DNS” you should see listings of DNS servers typically used for networks. Some won’t mind an individual connecting to them, others may even require granting permission. As an individual, you can connect to any “tier 2 DNS” servers as an alternate option to your ISP’s DNS. Some may be an improvement, others might be no improvement over your current DNS.
Google’s DNS are generally widely available for everyone to use no matter who they are… After all, if you use their servers you are contributing to their information on where everyone exists and what they are searching and connecting to, stuff they’d love to have.
If your ISP blocks access to someone else’ DNS, then you need to ask your ISP why they are doing it. It was once common to block DNS but it has been a long time since I’ve heard someone do something like that unless they <really> want to surveil you. And then, you’d need to consider using the Tor Browser Bundle in that case…
TSU
Hi
I use openDNS rather than google and over ride my ISP’s and it’s a simple entry into a network manager connection info to change if I’m away from home.
On 2014-06-14 00:06, ballsystemlord wrote:
> access sometimes. I don’t care how big it gets, several GIBs
Let me see…
255255255*255 entries.
That is 4228250625 lines…
At, say, 40…50 chars each, makes… more than 190 GiB.
A very gross approximation, though.
Even if it is half, and you have to update it daily. Can you download,
say, a 100 GB daily? And the way to obtain those lines is using “dig”,
which of course, queries DNS servers all over the world, and if DNS
queries do not work well at your site or are blocked… I do not see how
you could do it.
> is fine I’m
> just really sick of the message “Could not resolve host name” in
> fireofox, curl, wget, etc. If you could point out some way to filter
> what gets in /etc/hosts that would be even better.
you could run your one DNS server as cache.
Are you sure DNS queries are blocked?
For example, I get that problem when my ADLS download pipe is saturated
because I’m downloading something at full speed. Any DNS query simply
fails. It is probably my router fault, because it does not prioritize.
Maybe it would work better if I bypassed the router local DNS server.
–
Cheers / Saludos,
Carlos E. R.
(from 13.1 x86_64 “Bottle” at Telcontar)
Thanks guys, and yes, the US government library system of Sarasota blocks all DNS queries (they provide intranet so they are an ISP.) The library system, who most of the time provides my intranet blocks access to DNS, IRC, and all forms of UDP. Anything that is not port 443(https), 21(ftp), or 80(http). I have complained, politely, they don’t care. And I’m the only person I know that uses the library that understands what a DNS is. So, I get the “What is wrong with you! You are the ONLY one complaining! Aren’t we already too generous by letting you online at all!” I suspect that the admin, who I met with (and who told me that the problems I saw were nonexistent (reminds me of a certain senator who said that the American people could not figure out how to use a certain healthcare website,)) read a book that discussed minimizing the risk to your system is as easy as blocking everything. They also block access to the tor network and the tor project page (they have even gone so far as to block these forums from time to time as they are about 0x68acking!) I can change my DNS server through Network manager when I’m connected to a friends intranet and the library does not use a proxy, I checked by using curl with no proxy.
Thanks, I think I’ll set up my own DNS server. But, ignorant as I am, would the missing entries in their DNS propagate to my DNS?
By the way,robin_listas I’m not planning on having more then 5,000 entries in /etc/hosts (I normally don’t browse that much. SO, 5000*50 equals 2,500,000. But even assuming as many as you do, how would running my own DNS server change the size of the entries?
IMHO, running a DNS server yourself will not help as DNS is hierarchical and your DNS server needs access to the DNS servers higher up to get any information and that is the same blocking you have now (UDP port# 53)…
BTW I do not understand your company’s policy. What is the use of letting people using http, https and ftp on the Internet without DNS? Only very few will be able to type IP addresses using their browsers/tools.
And when nobody complains, it does not seem to be part of their work to go on the Internet. Except for private reasons and I do not see that the company should provide that.
On 2014-06-23 21:16, hcvv wrote:
>
> BTW I do not understand your company’s policy. What is the use of
> letting people using http, https and ftp on the Internet without DNS?
No, they force you to use the intranet DNS, and this one is filtered, so
that domains they do not like are effectively blocked, without actually
blocking the IPs.
The solution then is indeed have a local /etc/hosts file with the IPs
you need to access listed, and the list has to be limited. There are
problems: many web sites have frames an pictures and whatnot from other
sites, and you need to list them all. And many sites rotate or change
the actual IPs they point to.
A DNS of your own doesn’t help, because it has to populate by querying
outside, and this is blocked. The entries that your DNS caches have a
limited life, so after sometime they get erased.
If you could set up a tunnel via http to an outside site, that does the
real DNS serving for you, that would solve the problem - but I do not
know how to solve this.
–
Cheers / Saludos,
Carlos E. R.
(from 13.1 x86_64 “Bottle” at Telcontar)
In that case it is most likely that this technical action is based upon company rules. And thus that the OP’s whish to have those blocked DNS entries available and thus be able to circumvent the technical implementation of the company rules is something he/she should think over before the company finds out, because that will severely damage his/her relations with the company (loosing a job?).
On 2014-06-24 09:46, hcvv wrote:
>
> In that case it is most likely that this technical action is based upon
> company rules. And thus that the OP’s whish to have those blocked DNS
> entries available and thus be able to circumvent the technical
> implementation of the company rules is something he/she should think
> over before the company finds out, because that will severely damage
> his/her relations with the company (loosing a job?).
I understand it is not a job site, but a public library. LOL.
A public site must help the people, not hinder them.
And even in a job place, the job of the IT personnel includes helping
other people do their jobs. That is, the final goal of a business is not
to have Windows running on computers, or Linux, or whatever. They are
tools. It is what is done with the tools, and which the company sells or
gets paid for, that is the most important thing. It is thus the job of
the IT personnel to facilitate this goal, not to act god.
Thus if I tell IT that I need to use
http://forums.opensuse.org, which is currently blocked, they
have to give me access, because it blocks my work.
–
Cheers / Saludos,
Carlos E. R.
(from 13.1 x86_64 “Bottle” at Telcontar)
On 06/24/2014 10:13 AM, Carlos E. R. pecked at the keyboard and wrote:
> On 2014-06-24 09:46, hcvv wrote:
>> In that case it is most likely that this technical action is based upon
>> company rules. And thus that the OP’s whish to have those blocked DNS
>> entries available and thus be able to circumvent the technical
>> implementation of the company rules is something he/she should think
>> over before the company finds out, because that will severely damage
>> his/her relations with the company (loosing a job?).
> I understand it is not a job site, but a public library. LOL.
> A public site must help the people, not hinder them.
>
>
> And even in a job place, the job of the IT personnel includes helping
> other people do their jobs. That is, the final goal of a business is not
> to have Windows running on computers, or Linux, or whatever. They are
> tools. It is what is done with the tools, and which the company sells or
> gets paid for, that is the most important thing. It is thus the job of
> the IT personnel to facilitate this goal, not to act god.
Blaming the IT personnel is like shooting the messenger, They are only
doing what their boss tells them to do with said boss only doing what
rules to implement that come from upper management. I was that IT person
before I retired. If I was told to block certain sites I did so without
questioning the why.
Ken
On 2014-06-24 19:37, Ken Schneider wrote:
> On 06/24/2014 10:13 AM, Carlos E. R. pecked at the keyboard and wrote:
> Blaming the IT personnel is like shooting the messenger, They are only
> doing what their boss tells them to do with said boss only doing what
> rules to implement that come from upper management. I was that IT person
> before I retired. If I was told to block certain sites I did so without
> questioning the why.
In the OP case, apparently the IT person simply refuses to accept that
there is a problem.
And about shooting the messenger…
Once upon a time, on a telco company we were supervising old but crucial
equipment. We rigged the machines to send printer output, instead of to
paper, to a control room, via telnet (yes, telnet, not ssh). We also got
remote telnet access, instead of serial console on site.
Well, one morning we suddenly lost supervision. After pulling hairs for
hours, we found out that the local IT people (we were subcontractors)
had firewalled us, because our machines sent too much traffic.
It turned out that the routers we used for the conversion sent one full
TCP packet per single char the printer printed. We had to reconfigure
them to wait for a full line instead, which was easy to do after reading
the router documentation, and after learning that there was a problem.
But first, we had to go to top management to force the IT people to give
us access, so that we could implement the changes.
Those IT people could have contacted us (we were in the same room!
Gosh!) and told us that there was a problem, could we please do
something to reduce traffic. But no, they simply closed the tap and left
the entire national network without supervision for hours. If something
had broken in the machines, there could have been millionaire fines.
So, sorry, but I’m not very happy with the common IT person. There are
exceptions, but as a rule, I often had to work around them, in the job
place, to get done the work I was paid to do, and which they impeded.
–
Cheers / Saludos,
Carlos E. R.
(from 13.1 x86_64 “Bottle” at Telcontar)
On 2014-06-24 20:23, Carlos E. R. wrote:
> Those IT people could have contacted us (we were in the same room!
> Gosh!) and told us that there was a problem, could we please do
> something to reduce traffic. But no, they simply closed the tap and left
> the entire national network without supervision for hours. If something
> had broken in the machines, there could have been millionaire fines.
>
> So, sorry, but I’m not very happy with the common IT person. There are
> exceptions, but as a rule, I often had to work around them, in the job
> place, to get done the work I was paid to do, and which they impeded.
Forgot to say, that this particular group had a rule or motto they were
proud of: close everything first. Then wait for the complains. Then
consider opening some things. Few.
Sigh.
–
Cheers / Saludos,
Carlos E. R.
(from 13.1 x86_64 “Bottle” at Telcontar)
Getting back to the OP’s situation,
I do remember a few years ago I frequented a public library system in California once a month and discovered/determined whatever censoring system they were using, they did block all Linux related domains (openSUSE, RedHat, Ubuntu, CentOS, etc). I submitted my observation with full explanation how this was inappropriate and they promised to work on it. The following month, I still found Linux sites blocked, so I re-submitted. Another month later, and the problem was resolved. So, things sometimes to get done and maybe there is a censoring system commonly used by some library systems which is faulty in this way.
In any case, as I described the easiest workaround is to use Tor. My testing has verified that not only connections, but DNS requests are routed through the onion network (not using the ISP’s or local DNS). The drawback is that depending on time of day, the Tor network can be congested, and some protocols are prohibited but if you’re doing simple web browsing it’ll work with hopefully only slight DNS latency (I’ve determined at least where I am that after I’ve visited a Domain at least once, it’s cached and my connections are quick after that). Note that this implementation of tunneling DNS lookups is unique, typical VPNs and other tunneling typically use the locally configured DNS (not using the tunnel).
TSU
On 2014-06-27 10:06, tsu2 wrote:
>
> Getting back to the OP’s situation,
>
> I do remember a few years ago I frequented a public library system in
> California once a month and discovered/determined whatever censoring
> system they were using, they did block all Linux related domains
> (openSUSE, RedHat, Ubuntu, CentOS, etc). I submitted my observation with
> full explanation how this was inappropriate and they promised to work on
> it. The following month, I still found Linux sites blocked, so I
> re-submitted. Another month later, and the problem was resolved. So,
> things sometimes to get done and maybe there is a censoring system
> commonly used by some library systems which is faulty in this way.
Yiks.
> In any case, as I described the easiest workaround is to use Tor.
But the OP said they also block tor. :-?
–
Cheers / Saludos,
Carlos E. R.
(from 13.1 x86_64 “Bottle” at Telcontar)
You know robin listas, you seem to be one of the only people who reads my posts… (is it my posts or others impatience…)
Yes, they block tor, DNS, etc. If you’ve been taking note, then you would notice that I’ve been interested in the kernel for some time, but guess what, IRC is blocked too, yipe! (if you don’t want 900 messages per week in your email box then you use IRC to kernel.org to talk to people.) I’m not quite sure why they block what they block, it seems almost haphazard (but the blocking of ports is defiantly preplanned.)
I guessed that if I set up a DNS server on my machine then I’d be at a much greater risk for getting incorrect routing info because I visit more then one access point. I’d also have to deal with theirs and others DNS servers not having the entry at all.
So, thanks for the help I’ll populate /ect/hosts. Oh, I should have mentioned before that I don’t need to download all the DNS info every time I connect, just when the info becomes out of date.
On 2014-07-09 21:36, ballsystemlord wrote:
>
> You know robin listas, you seem to be one of the only people who reads
> my posts… (is it my posts or others impatience…)
Well, I understand the situation.
It is not a job place situation, in which case you are bound hand and
legs. There you have to be very careful with the legal situation. Or
firing situation, rather.
> Yes, they block tor, DNS, etc. If you’ve been taking note, then you
> would notice that I’ve been interested in the kernel for some time, but
> guess what, IRC is blocked too, yipe! (if you don’t want 900 messages
> per week in your email box then you use IRC to kernel.org to talk to
> people.)
900 per week is not that much 
Just use an account such as gmail, and everything goes there. You do not
even need to download them to your machine if you do not want to.
Then there is gmane. If they support the list you want, you access it
via nntp instead of email, which allows you to download only the email
bodies you want to read (but all the headers, but you can filter by
number or date).
Provided your library does not block them or nntp, which they might, both.
See the wikipedia for an explanation.
> I’m not quite sure why they block what they block, it seems
> almost haphazard (but the blocking of ports is defiantly preplanned.)
> I guessed that if I set up a DNS server on my machine then I’d be at a
> much greater risk for getting incorrect routing info because I visit
> more then one access point.
No, that’s not an issue. You just tell your server to query different
servers. But you simply can not set it up on that library, because they
block it.
Notice that “routing info” is not related to “dns”.
> I’d also have to deal with theirs and others
> DNS servers not having the entry at all.
Not only that, but that they block you querying another source.
> So, thanks for the help I’ll populate /ect/hosts. Oh, I should have
> mentioned before that I don’t need to download all the DNS info every
> time I connect, just when the info becomes out of date.
It will be very problematic, I’m afraid.
A solution would be, if you have access to a machine outside, and to
which you can access somehow - for instance, via http -, to then set up
a tunnel via that protocol and then have it serve your dns queries - or
perhaps to use it as an http redirector to another place, using apache
in that machine to do it. I know the latter is possible, but not how to
do it.
if you don’t have such a machine outside, then you could hire a server
at a hosting place, and use it for the purpose. But you need remote
access to it, so to administer and set it up. I also do not know exactly
how to do it.
But I would seek another library, if possible…
–
Cheers / Saludos,
Carlos E. R.
(from 13.1 x86_64 “Bottle” at Telcontar)
You seem to think that we ignore you, but more people are reading. We not normaly start telling you that we have read it but that we can not add any usefull remark. That would swamp your thread with useless posts.
We all have pitty that you seem to be bound to such a horrible place to reach the internet. For me it is the same as not offering Internet access at all. But I do not know of any way to add all DNS resolves that succeed on your system, because of some program asked for it, to put them automaticaly in your /etc/hosts.
On 07/10/2014 06:26 AM, hcvv wrote:
>
> @ballsystemlord,
>
> You seem to think that we ignore you, but more people are reading. We
> not normaly start telling you that we have read it but that we can not
> add any usefull remark. That would swamp your thread with useless posts.
>
> We all have pitty that you seem to be bound to such a horrible place to
> reach the internet. For me it is the same as not offering Internet
> access at all. But I do not know of any way to add all DNS resolves that
> succeed on your system, because of some program asked for it, to put
> them automaticaly in your /etc/hosts.
No easy way, but one could write a program that monitored the network traffic
using tcpdump, and matched every nameserver lookup with the response, and added
every new one to the hosts file. That would certainly not be easy. In addition,
if you did it badly, you would consume the entire computer every time you opened
a new web site with a lot of ads.