How to prohibit IP from directly accessing HTTPS

The following settings are set in nginx:

server {
    listen 80;
    listen 443 ssl spdy;
    root /data0/web/domain.com;
    server_name domain.com *.domain.com;
    index index.html index.htm index.php;

    location / {
    }
}

##default
server {
    listen 80 default;
    listen 443 default;

    server_name _;

    root /data0/web/empty;

    location / {
        return 500;
    }
}

Set as above, access https://ip. SSL doesn't work. Even if you visit https://domain.com, you cannot access it.